What pic-local-harness Actually Is: An Engineer’s Guide to Local AI Security Evaluation

If you are an engineer who is new to AI systems, the first thing to understand is this: pic-local-harness is not “just another script that sends prompts to a model.” It is a local test harness for evaluating how AI-enabled applications behave when they are exposed to adversarial inputs such as prompt injections, jailbreaks, or manipulative instructions.

The core idea is simple:

• AI applications are not only software systems
• they are software systems that delegate part of their behavior to a probabilistic model
• that delegation creates a new attack surface
• you need a systematic way to test that surface

This document explains that concept in engineering terms. It covers:

1. what problem this kind of harness solves
2. what happens under the hood, step by step
3. why this matters for local applications
4. how the same pattern fits into a production-quality application in the cloud
5. what processes make such a framework self-sustaining over time
6. how pic-local-harness implements those ideas locally

This is deliberately detailed. The goal is not a quick start. The goal is understanding.

1. Why AI applications need a harness at all

In a traditional application, the control flow is mostly explicit.

• request comes in
• validation runs
• business logic runs
• response comes out

If the app is buggy, you debug deterministic code paths.

In an AI application, one part of the control flow is no longer fully explicit. At some point the application asks a model to generate or transform text, rank content, select tools, or synthesize an answer. That means the application is no longer only executing your code. It is executing:

• your code
• your prompt design
• your retrieval design
• your output filtering
• your model behavior
• your model provider’s safety behavior

That stack is much harder to reason about.

1.1 The attack surface changes

When an app uses an LLM, the attacker’s job is often no longer “inject SQL” or “smuggle shell characters.” The attacker’s job becomes “get the system to follow my malicious instruction instead of the application’s intended instruction.”

That can happen through:

• direct prompt injection
• indirect prompt injection through retrieved content
• tool-use coercion
• role confusion
• instruction override attacks
• context-window poisoning
• refusal evasion
• system prompt extraction attempts

The important part is this: many of these attacks do not break a parser or crash a service. They exploit the semantic behavior of the application.

That means normal unit tests and integration tests are not enough.

1.2 Manual testing is not sufficient

Without a harness, teams usually do one of these:

• manually paste attack prompts into the app UI
• run a few ad hoc curl commands
• depend entirely on model-provider safety claims
• wait until production traffic reveals issues

All of those approaches are weak.

Manual testing fails because:

• it is not reproducible
• it is not observable enough
• it does not scale
• it produces weak evidence
• it does not tell you what changed between versions

A harness exists to fix those shortcomings.

2. The conceptual model behind pic-local-harness

At a high level, pic-local-harness is a local evaluation loop.

It takes four inputs:

1. a target application
2. a model endpoint
3. a corpus of test prompts
4. a scoring policy

And it produces:

1. request/response artifacts
2. structured logs
3. pass/fail summaries
4. forensic traces for every tested case

That sounds simple, but each of those parts matters.

2.1 The target application

The target is not the model directly. The target is the application that wraps the model.

That distinction is critical.

In production, users do not talk to your raw model endpoint. They talk to:

• your chat API
• your RAG layer
• your session/auth layer
• your application-specific prompt templates
• your output shaping code
• your refusal and filtering logic

If you only test the base model, you do not test the real system.

2.2 The model endpoint

The harness uses a model endpoint because most modern AI apps depend on one. In local mode, the harness standardizes on OpenAI-compatible HTTP because that is the most practical common surface across many local backends.

That means the harness can usually work with:

• vLLM
• LM Studio
• local gateways
• OpenAI-compatible proxies
• some Ollama setups exposed through compatible shims

This is not because OpenAI is the only way to do things. It is because engineering time is finite and a common protocol reduces friction.

2.3 The corpus

The corpus is a collection of test prompts. Some are benign. Some are malicious. Some are ambiguous.

A good corpus lets you ask questions like:

• does the app reject obvious attack prompts?
• does it over-block normal requests?
• does it behave differently when inputs are phrased differently?
• do changes in prompt templates or retrieval policies improve or worsen outcomes?

The corpus is what turns hand-wavy intuition into repeatable evaluation.

2.4 The scoring policy

The harness must classify results somehow. It needs a definition of success and failure.

That can be:

• status-code based
• regex-based
• adapter-defined
• or eventually something richer

The point is not that the classification is perfect. The point is that it is explicit, reviewable, and versionable.

3. What happens under the hood, step by step

This is the most important section if you want to understand the system, not just use it.

Step 1: The operator selects a profile

A profile is a YAML document that defines the local test topology.

It tells the harness things like:

• which local model endpoint to use
• which application(s) to test
• whether the application is already running or must be started
• which corpus file to load
• how many prompts to sample
• which detector logic to apply
• what thresholds count as failure

In other words, the profile turns “run the tests” from a vague instruction into a concrete, reproducible execution plan.

Step 2: The harness resolves the profile

Inside the code, profile resolution means:

• loading the YAML
• applying inheritance if extends is used
• merging nested settings
• expanding environment variables
• constructing strongly shaped internal config objects

This is not just convenience. It matters because AI testing becomes unmanageable when configuration is implicit or scattered.

The resolved configuration is written to run.json for the exact run, so there is no ambiguity later about what the harness actually used.

Step 3: A run directory is created

The harness creates a unique run directory.

This is the execution container for observability.

It stores:

• the resolved config
• structured events
• readable console logs
• process logs
• request/response artifacts
• per-app summaries
• per-case results

This is one of the core architectural decisions. Without a durable run directory, debugging is largely guesswork.

Step 4: Logging is initialized

The harness starts two logging streams:

1. structured JSONL events
2. a human-readable console log

This dual format exists for a reason.

• JSONL is good for tooling, filtering, and correlation
• plain log text is good for fast human understanding

A mature AI evaluation framework needs both.

Step 5: The model endpoint is defined

The harness does not invent a model. It binds to one through profile configuration.

For local usage, that generally means:

• a base_url
• an optional API key
• a chat model name
• an embedding model name

This lets the harness test the app against your local inference stack rather than a cloud-managed model.

That matters because local model choice is often part of product engineering itself. You may want to compare:

• a small fast local model
• a larger local model
• two different prompt configurations
• two different safety wrappers

A harness gives that comparison structure.

Step 6: The target application is attached or started

Every app has a service definition.

There are two modes.

Attached mode

The harness assumes the service is already running.

This is useful when:

• you started the app manually
• you are already debugging it in another terminal
• the app startup is complex and you want control over it

Managed mode

The harness starts the app process itself.

This is useful when:

• you want repeatability
• you want one command to drive the whole test flow
• you want the harness to capture stdout/stderr automatically

In either mode, the harness uses healthchecks to decide whether the app is actually usable.

That is important. Many systems “start” but are not yet ready.

Step 7: Application-specific environment is prepared

The harness does not assume all applications use the same model variable names or startup rules.

Instead, each app adapter prepares the environment it needs.

For example:

• one app might expect LLM_BASE_URL
• another might expect MODEL_BASE_URL
• one app might need a local auth bypass or bootstrap path
• another might need an API key header

This is exactly why app adapters exist.

A generic HTTP sender is not enough. Real applications have real contracts.

Step 8: The corpus is loaded

The harness loads a local JSONL corpus.

For each record, it validates required fields like:

• text
• label
• source

Then it can apply filters such as:

• labels only
• source subsets
• tier subsets
• deterministic sampling

Deterministic sampling matters more than many people realize. If you are debugging a problem, you need to know that sample size 50 with seed 17 today means the same 50 test cases tomorrow.

Step 9: Each case is executed

This is the actual evaluation loop.

For each corpus row:

1. the harness assigns a case id
2. it logs that the case started
3. it asks the app adapter to build the request
4. it sends the request to the local app
5. it captures raw response data
6. it asks the adapter to normalize the response
7. it applies detector logic
8. it decides blocked, bypassed, or error
9. it writes a case artifact to disk
10. it emits structured completion events

This is where most of the practical value comes from.

The system is not only deciding whether the app passed. It is preserving the evidence for why it decided that.

Step 10: Request artifacts are saved

Per-case request artifacts are critical.

Each one answers questions like:

• what exact payload was sent?
• what URL was used?
• what headers were applied?
• what status code came back?
• what body came back?
• how did the adapter interpret it?
• why did the detector mark it blocked or bypassed?
• how long did the case take?

If you have ever debugged AI behavior from incomplete logs, you know why this matters.

Step 11: App-level summaries are computed

After all cases for an app are complete, the harness computes aggregate metrics such as:

• total cases
• blocked cases
• bypassed cases
• errored cases
• ASR
• threshold pass/fail

This is the level where engineering decisions get made.

• Did a prompt-template change help?
• Did a new local model regress?
• Did a retrieval tweak increase over-refusal?

Without aggregate metrics, all you have are anecdotes.

Step 12: The run is finalized

At the end, the harness:

• writes overall summaries
• emits run completion events
• stops managed processes if needed

At that point the run directory becomes the permanent record of what happened.

4. Why this is useful for local applications

People often assume security evaluation matters only once they have a cloud deployment. That is backwards.

Local evaluation is where you get cheap feedback.

4.1 Fast iteration

If you can run the harness locally, you can validate changes immediately after modifying:

• prompt templates
• retrieval ranking
• output filtering
• model routing
• refusal logic
• session/auth behavior

That is much faster than pushing everything to staging and waiting for CI or cloud resources.

4.2 Lower cost

Running the loop locally lets you avoid burning money on:

• cloud inference during development
• staging infrastructure just for routine checks
• repeated remote deployments for minor changes

This does not eliminate the need for production-like validation, but it dramatically reduces when you need it.

4.3 Better debugging

Local execution gives you easier access to:

• app process logs
• debugger attachment
• request traces
• environment inspection
• iterative re-runs

In cloud environments, debugging often becomes log archaeology. Locally, you can move much faster.

4.4 Better model experimentation

A local harness is especially useful when model choice itself is changing.

You can compare:

• model A vs model B
• different quantizations
• different safety prompts
• different retrieval thresholds
• local-only vs cloud-like prompt strategies

This is valuable both for performance engineering and for safety engineering.

5. How this maps to a production-quality cloud application

A local harness is not the final production system, but it models the same evaluation logic you want in production.

A mature production setup generally contains the same conceptual stages.

5.1 Corpus management

In production, the corpus usually lives in its own governed repository or dataset pipeline.

Typical practices include:

• versioning the corpus
• tagging records by attack type
• separating benign from malicious samples
• curating new records from incidents
• promoting stable benchmark sets

The local harness consumes a local corpus, but in production the corpus becomes a controlled testing asset.

5.2 Environment tiers

A production-quality system usually has at least three tiers:

• local/dev
• staging/pre-production
• production

The harness pattern fits all three, but the goals differ.

Local

• rapid iteration
• low cost
• developer visibility

Staging

• contract validation against realistic infrastructure
• auth and deployment checks
• app-level regression detection

Production

• limited canary checks
• passive telemetry
• incident-driven replay
• periodic benchmark execution in controlled windows

5.3 Production adapters and contracts

In a real production environment, adapters often become more important, not less.

That is because cloud apps add more complexity:

• auth tokens
• session minting
• rate limits
• environment-specific routing
• gateway behavior
• retries
• feature flags
• fallback models

The harness approach scales because it treats each target as an adapter-defined contract rather than pretending all apps are the same.

5.4 Metrics and release gates

In production-quality engineering, the harness is often part of a broader evaluation gate.

Examples:

• block a deployment if ASR exceeds threshold
• run a sample corpus on every PR
• run a larger corpus on release branches
• compare results against a baseline run
• page someone if canary behavior regresses sharply

The local harness is the local expression of that same pattern.

6. What “self-sustaining” means for this kind of framework

A framework like this becomes self-sustaining when it no longer depends on heroic manual effort to stay useful.

That requires process, not just code.

6.1 Corpus maintenance process

A self-sustaining framework needs a way to keep the corpus current.

That usually means:

• adding new attack samples from incidents
• curating new prompt variants from research
• removing or retagging low-quality examples
• keeping a stable benchmark subset for longitudinal comparison

Without this, the framework decays into a stale benchmark.

6.2 Adapter maintenance process

Applications evolve. Routes change. Response formats change. Auth flows change.

So there must be a process to keep adapters aligned with real applications.

That means:

• owning adapter updates alongside app changes
• validating adapter assumptions in staging or local smoke tests
• updating runbooks when contracts change

If adapter ownership is vague, the framework drifts out of usefulness.

6.3 Baseline management

A serious evaluation framework needs a notion of “normal.”

That can be:

• the last known good run
• a release candidate baseline
• a pinned model baseline
• a fixed local-model baseline

Then you can ask meaningful questions:

• did ASR increase?
• did refusal behavior become too aggressive?
• did latency change materially?
• did one model improve while another regressed?

6.4 Scheduled validation

Even for local-first tooling, self-sustaining systems benefit from routine execution.

Common patterns:

• local ad hoc runs during development
• nightly sampled staging runs
• weekly broader corpus checks
• release-time full benchmark runs

The point is to prevent evaluation from becoming optional or forgotten.

6.5 Incident feedback loop

One of the strongest uses of a harness is turning incidents into test cases.

If an issue occurs in production, the right response is not just to patch the app. It is also to:

1. capture the failing input pattern
2. normalize it into a reusable corpus sample
3. add it to a maintained suite
4. ensure future versions are tested against it

That is how a harness becomes a real engineering asset rather than a demo tool.

6.6 Ownership and governance

A self-sustaining framework needs owners.

Typically that means explicitly assigning responsibility for:

• corpus quality
• adapter correctness
• threshold policy
• runbook accuracy
• baseline review
• incident replay procedures

If everyone assumes someone else owns the framework, it will rot.

7. How pic-local-harness is put to use in a real engineering workflow

The production-quality pattern usually looks like this.

7.1 During development

An engineer changes:

• a prompt template
• a refusal heuristic
• retrieval ranking
• an app route
• a local model configuration

Then they run:

• plh resolve-config
• plh doctor
• plh run --profile <small sample profile>

They inspect:

• summary output
• one or two suspicious case artifacts
• process logs if startup was unstable

7.2 Before merge

A stronger sampled run is executed.

Goals:

• catch obvious regressions
• verify auth and app contracts still hold
• make sure a local improvement did not create a new failure class

7.3 Before release

A broader benchmark set is executed in a more production-like environment.

Goals:

• validate behavior under the real deployment topology
• compare against the current release baseline
• verify thresholds remain acceptable

7.4 After incident

A case artifact or live issue is converted into a test sample.

Then the framework becomes the regression-prevention mechanism.

That is what makes the system self-improving over time.

8. Why the framework is especially valuable for local applications

The strongest benefit of a local harness is not only cost. It is control.

With a local setup, you control:

• process startup order
• model version selection
• environment variable injection
• request capture
• corpus size
• replay speed
• debugging depth

That lets you answer questions much faster.

For example:

• Did the app fail because the model changed?
• Did it fail because the adapter is wrong?
• Did it fail because auth was misconfigured?
• Did it fail because a retrieval change altered the context?
• Did it fail because the detection regex is too weak?

A local harness narrows that uncertainty quickly.

9. What this framework does not magically solve

It is important to be precise here.

A harness does not automatically give you perfect AI security.

It does not:

• prove your app is safe
• replace careful prompt design
• replace auth controls
• replace output filtering
• replace production monitoring
• replace cloud parity testing

What it does is give you a disciplined evaluation loop.

That loop is necessary. It is just not sufficient by itself.

10. The practical mental model to keep

If you are new to AI systems, the right mental model is this:

• your AI app is a software system with a non-deterministic subsystem
• that subsystem changes the threat model
• therefore you need a repeatable evaluation framework
• the framework needs both corpus inputs and operational observability
• local execution is the cheapest place to build that discipline
• cloud or production validation then becomes a later extension of the same pattern

pic-local-harness is the local implementation of that pattern.

It lets you treat AI behavior as something you can systematically evaluate rather than something you only “try out.”

11. How to put this into use immediately

If you want to use the framework correctly, use this order:

1. define a profile for one app
2. point it at one local model endpoint
3. run plh resolve-config
4. run plh doctor
5. execute a tiny sample corpus
6. inspect one or two case artifacts in detail
7. tighten detector rules if needed
8. increase sample size only after the small run is stable
9. add the same pattern to the second app
10. later, mirror the same evaluation logic in staging or production release gates

That is the practical path from local experimentation to production-quality discipline.

12. Final takeaway

The real value of a framework like this is not that it sends prompts. Any script can do that.

The real value is that it gives an engineering team:

• repeatability
• observability
• evidence
• comparison points
• a path from local validation to production governance

That is why this concept matters.

And that is why a local-first harness is worth building before you spend time and money turning every test into cloud infrastructure.