Evaluating Senior Engineers in the Age of AI

Recently, I attended an interview for a senior or staff engineering role. The conversation was professional, but it left me thinking about how we evaluate experienced engineers today.

A meaningful portion of the discussion focused on questions like:

• Have you used SonarQube?
• Which code review tool do you use?
• Which pipeline tool have you worked with?
• Which monitoring platform do you prefer?

These are fair questions. Tools matter. A senior engineer should understand the ecosystem around code quality, CI/CD, observability, security, and delivery.

But after more than 15 years of building enterprise systems, MES platforms, mobile applications, APIs, integrations, distributed systems, and production-critical solutions, I have started to wonder whether tool-based questions are enough to reveal real engineering experience.

Tools Are Useful, but They Are Not the Signal

Tools change constantly.

A team using Azure DevOps today may move to GitHub Actions tomorrow. A company using SonarQube may switch to another code quality platform next year. Monitoring stacks, security scanners, deployment platforms, cloud services, and development workflows keep evolving.

Experienced engineers should be able to learn tools. That is part of the job.

The deeper question is not whether someone has clicked through a particular dashboard before. The deeper question is whether they understand the problem the tool is supposed to solve.

For example:

• Code quality tools exist to reduce defects, maintainability risks, and hidden complexity.
• CI/CD tools exist to create reliable, repeatable delivery.
• Monitoring tools exist to improve visibility into system behavior.
• Security tools exist to reduce risk before and after deployment.

A tool name can tell you what someone has been exposed to. It does not always tell you how they think.

What Experienced Engineers Actually Get Paid For

At senior and staff levels, the value of an engineer is not limited to writing code or remembering syntax. The real value often appears when the situation is ambiguous, incomplete, or under pressure.

That is where better interview conversations can happen.

I would rather understand how an engineer approaches questions like:

• How do you handle a production outage when the root cause is unclear?
• How do you design a system that can scale without over-engineering it on day one?
• How do you troubleshoot a problem nobody on the team has seen before?
• How do you make technical trade-offs when every option has a cost?
• How do you mentor a team through a difficult delivery?
• How do you balance delivery pressure with long-term maintainability?
• How do you decide when to refactor, when to rewrite, and when to leave working code alone?

These conversations reveal judgment.

They show whether someone can reason from first principles, communicate trade-offs, protect production systems, guide teams, and adapt when assumptions change.

That is much harder to fake than a list of tools.

AI Has Changed the Value of Memorization

There is another reality we should acknowledge: the industry has changed.

Years ago, remembering syntax, APIs, configuration details, and framework behavior had significant value. Then developer tooling improved. IntelliSense, static analysis, better documentation, package discovery, templates, and cloud consoles all reduced the need to remember everything manually.

Now we have AI-assisted development.

This does not mean fundamentals no longer matter. They absolutely do.

A developer who does not understand data structures, transactions, concurrency, network boundaries, security, failure modes, or system design will still make poor decisions even with the best tools available.

But the value of pure memorization is decreasing.

The value of problem-solving is increasing.

The value of architecture is increasing.

The value of engineering judgment is increasing.

The value of adaptability is increasing.

The best engineers are not the ones who remember every method signature. They are the ones who can understand a problem, break it down, ask the right questions, use modern tools responsibly, validate the output, and deliver a maintainable solution.

Better Ways to Evaluate Senior Engineers

If engineers are expected to evolve with technology, interview processes should evolve as well.

For experienced roles, we can go beyond tool checklists and memory-based quizzes. Some better evaluation formats are:

1. Discuss a Realistic Case Study

Give the candidate a system scenario with constraints.

For example: a manufacturing workflow system has intermittent delays, database load is increasing, and users are reporting inconsistent status updates across mobile and web applications.

Ask the candidate how they would investigate it.

The goal is not to find one perfect answer. The goal is to observe how they clarify requirements, identify risks, reason about dependencies, and separate symptoms from root causes.

2. Review Architecture Decisions

Present a design and ask what could go wrong.

A strong engineer should be able to discuss reliability, scalability, observability, data consistency, deployment complexity, operational cost, and team capability.

They should also be able to say, “This design may be good enough for the current stage,” instead of automatically recommending the most complex architecture.

That restraint is part of seniority.

3. Debug a Production-Like Issue

Instead of asking only theoretical questions, give the candidate logs, metrics, a failing workflow, or a short code path with unclear behavior.

Ask them to think aloud.

How do they narrow the problem? What assumptions do they test first? Do they jump to conclusions? Do they consider recent deployments, data changes, infrastructure, concurrency, integration failures, and user behavior?

Debugging reveals experience very quickly.

4. Review Code Together

A collaborative code review can show far more than a syntax quiz.

It reveals how the engineer thinks about readability, maintainability, testability, naming, boundaries, error handling, performance, and security.

It also reveals communication style. Senior engineers do not only find problems; they help teams improve without creating fear or unnecessary friction.

5. Explore Trade-Offs

Most real engineering decisions are not between good and bad. They are between competing costs.

Speed versus maintainability.

Consistency versus availability.

Flexibility versus simplicity.

Build versus buy.

Short-term delivery versus long-term ownership.

A senior interview should create space for these discussions because experienced engineers spend a lot of time operating in trade-offs.

What a Strong Senior Interview Should Feel Like

A great interview should not feel like a quiz.

It should feel like two professionals exploring whether they can solve difficult problems together.

That does not mean lowering the bar. In fact, it raises the bar.

It is easier to ask whether someone has used a tool. It is harder to evaluate how they think under uncertainty.

But senior engineering work is full of uncertainty.

The best interviews should test the work that actually matters:

• Can this person reason clearly?
• Can they design systems with constraints?
• Can they debug unfamiliar problems?
• Can they communicate trade-offs?
• Can they use tools without becoming dependent on them?
• Can they guide others through complexity?
• Can they protect production while still delivering business value?

Those are the signals worth looking for.

Closing Thought

Tool experience has value, but it should not become the center of senior engineering evaluation.

In the age of AI-assisted development, the strongest engineers will not be defined by how much syntax they remember or how many tools they can list.

They will be defined by how well they understand problems, make decisions, adapt to change, and help teams deliver reliable systems.

If our engineering practices are evolving, our interview practices should evolve too.

The real question is not, “Which tools have you used?”

The better question is, “How do you think when the answer is not obvious?”