Evaluation

Purpose of Evaluation

Evaluation in Blankstate is a critical process to ensure that the frameworks defining Operational Excellence (OE) – namely, Blueprints and their constituent Protocols – are accurate, effective, and aligned with organizational goals. It also involves understanding and validating the performance of the underlying AI models that power the analysis.

Blueprint Evaluation

Evaluating a Blueprint involves assessing its overall structure, the relevance and coverage of its included Protocols and Metrics, and its expected performance against various operational scenarios. This is primarily done through the blueprint.blankstate.ai interface.

• Structure and Alignment: Reviewing the selection of Protocols and Metrics to ensure they collectively cover the desired aspects of Operational Excellence defined by the Blueprint's purpose and taxonomy distribution.
• Metric Coverage: Using EVA's capabilities (see Metrics documentation) to verify that all required Protocols for included Metrics are present and active within the Blueprint draft.
• Simulation and Testing: The blueprint.blankstate.ai interface offers sandbox environments or simulation tools to test a draft Blueprint. You can run sample data or scenarios through the Blueprint's defined Protocols and Metrics to observe the resulting scores and OEI values, helping refine configurations before deployment.
• Replay Evaluation Mode: The Replay application provides a dedicated Evaluation Mode. This allows you to apply a draft Blueprint to specific historical documents or datasets to see how it performs in a realistic context, identifying potential gaps or inaccuracies in Protocol scoring or Metric calculation.

Protocol Evaluation

Evaluating individual Protocols is crucial for ensuring the accuracy and reliability of the "sensors" that feed data into your Metrics and overall Blueprint. This process is significantly enhanced by the automated Quality Indicators (QIs) calculated during the AI-Assisted Protocol Creation workflow.

• Quality Indicators (QIs): During creation (especially via the AI-Assisted workflow in blueprint.blankstate.ai), Protocols are automatically assessed and labeled with QIs such as Completeness, Specificity, Actionability, Metric Alignment, Source Evidence, and IBF Compatibility.

These QIs provide a summarized assessment of the Protocol's expected performance and characteristics without requiring manual deep dives into its complex IBF definition. They are calculated using automated analysis, including leveraging the underlying Core AI Model.

• Review via QIs: Mind Architects and administrators primarily evaluate Protocols by reviewing these automatically calculated QIs within the blueprint.blankstate.ai interface. Low scores on certain QIs (e.g., Low Specificity) prompt refinement of the Protocol definition.
• Refinement: Based on QI feedback or observed performance in Blueprint simulations or Replay evaluations, Protocols can be refined by adjusting their high-level definition or, for advanced users, directly editing the IBF definition.

Blueprint Evaluation (IBF & Consensus AI)

Blankstate's analytical power is derived from its proprietary AI models, primarily the Intention Blended Framework (IBF), Core Self-supervised Model and components of Consensus AI (EVA). While users do not directly interact with model training, understanding their evaluation is key to trusting the system's outputs.

• IBF Core AI Model: This model is the engine that performs the nuanced scoring of interactions (Stream) or content (Replay) against Protocol definitions.

The automated calculation of Protocol Quality Indicators is a direct application of our modeling capabilities, providing an indirect form of model validation visible to all.

• Consensus AI (EVA): Components of Consensus AI are used for tasks like AI-Assisted Protocol Creation (generating IBF definitions), Entity Mapping in Replay, Rationale Generation, and Report Generation. Evaluation of these components focuses on the quality, relevance, and accuracy of their generated outputs through internal testing and user feedback mechanisms.

Evaluation's Impact on Metrics and OEI

The outcome of Blueprint and Protocol evaluation directly impacts the reliability and meaning of the Metrics and the overall Operational Excellence Index (OEI).

• Metric Accuracy: Metrics derive their values from aggregated Protocol scores. If Protocols are not well-defined or evaluated (e.g., low Specificity, low Completeness), the Metrics they feed will be less accurate or meaningful.
• OEI Reliability: The OEI is a composite score based on the performance of active Metrics. A Blueprint containing poorly evaluated Protocols or incorrectly configured Metrics will result in an OEI that does not accurately reflect the true operational state.
• Iterative Improvement: Evaluation is an iterative loop. Insights gained from observing Metric trends and OEI fluctuations in Stream (via EVA Control Center) or analyzing results in Replay can inform the need to re-evaluate and refine the underlying Blueprint, Protocols, or Metric configurations using the tools in blueprint.blankstate.ai.