Okta is hiring a Principal ML Engineer SecureAI in Toronto!
Okta is hiring a Principal Machine Learning Engineer for its SecureAI organization in Toronto, specifically within the Agent Access Policies sub-team. The group is building an identity security platform for the "agentic era" — infrastructure that lets enterprises deploy autonomous AI agents while keeping every access decision authenticated, authorized and audited. The core problem is moving authorization away from static, rule-based policy toward dynamic, model-driven security: real-time threat inspection, evaluation of what an agent actually intends to do, and behavioural analysis that keeps agents inside defined bounds. This is a senior individual-contributor role with mentoring responsibilities, working alongside engineers, product managers and designers on a greenfield product area, including standards work around agentic identity and protocols such as MCP.
Key responsibilities
- Design and ship intent-based enforcement that checks runtime agent requests against their stated purpose
- Apply LLM reasoning and prompt parsing to interpret prompts, tool payloads and agent intent in real time
- Embed low-latency inference and semantic evaluation into the API gateway request path
- Use embeddings, vector search and zero-shot classification to score alignment between intent and executed actions
- Build confidence-scored decision engines that feed verification results into policy frameworks such as Cedar
- Define evaluation benchmarks, prompt-injection defenses and guardrails against bypasses and false positives
- Architect scalable ML and generative AI systems spanning retrieval, inference and evaluation pipelines
- Tune prompting, context retrieval and RAG workflows for accuracy, safety and efficiency
- Create automated evaluation pipelines covering model quality, correctness, groundedness and safety in production
- Enforce schema validation, structured output and guardrails for compliant AI outputs
- Mentor and coach engineers to grow the team
Key requirements
- 10+ years of software development experience with strong Python skills; Go or TypeScript a plus
- Hands-on applied ML experience, from feature engineering through training and fine-tuning
- Experience with generative AI platforms such as AWS Bedrock, OpenAI or Anthropic
- Deep understanding of RAG, embeddings and knowledge-base workflows
- Hands-on work with LiteLLM, LangGraph, LangChain, LlamaIndex, MCP or similar agent frameworks
- Familiarity with ML frameworks (FastAPI, PyTorch, TensorFlow, Spark ML) and orchestration tools like Airflow
- Track record defining evaluation metrics, pipelines and feedback loops for ML/GenAI systems
- Ability to drive greenfield initiatives with product and engineering partners through ambiguity
- Bachelor's or Master's in Computer Science or a related field
- Nice to have: AI systems integrated with identity, authentication or security products; ethical AI, model risk or compliance exposure; synthetic data generation or LLM-as-a-judge methods
- Seniority: Principal
- Work mode: Hybrid
- Salary: $192,000 - $264,000 CAD (stated as the annual base range for San Francisco Bay Area candidates)
- Key skills: Python, Go, TypeScript, Applied machine learning, Generative AI, AWS Bedrock, OpenAI, Anthropic, RAG, Embeddings, Vector search, LangGraph, LangChain, LlamaIndex, MCP
Want more jobs?
Join our mailing list to receive updates about new data science and Principal ML Engineer SecureAI jobs.
