21 Evaluating and Refining Outputs

• Legal Research and Analysis — Focuses on verifying citations, ensuring the currency of the law, and checking the completeness of the analysis.
• Contract Drafting and Review — Emphasises consistency across terms, completeness of provisions, and jurisdictional compliance.
• Due Diligence Document Review — Concentrates on completeness, consistency across documents, and identification of critical provisions.
• Client Communication and Legal Advice — Highlights ethical compliance, relevance, and appropriate explanation of complex concepts.
• Regulatory Compliance Analysis— Focuses on the currency of regulations, jurisdictional accuracy, and comprehensive coverage of requirements.
• Legal Education and Knowledge Management — Emphasises accuracy while simplifying concepts, audience appropriateness, and balanced presentation.

 

To assist in evaluating AI-generated content, the following section highlights the essential actions and processes to consider when reviewing content.

1. Cross-Reference with Authoritative Sources

Key Action: Verify all legal information, case citations, and statutory references against established legal databases.

Process:

• Identify all citations, legal principles, and statutory interpretations in the AI output.
• Look up each reference in Westlaw, LexisNexis, or official court/legislative websites.
• Verify that the AI’s interpretation aligns with the actual text and context of the cited source.
• Check that the quoted material is presented in the proper context.

Critical Questions:

• Do all cited cases and statutes exist?
• Are the holdings and principles accurately represented?
• Has the model fabricated or misattributed any legal authority?
• Does the interpretation align with reputable secondary sources?

Practice Tip: Create a verification checklist for each document type with standard references that must be verified.

2. Consistency Check

Key Action: Ensure that the output maintains logical and legal consistency throughout.

Process:

• Review the document for contradictory statements.
• Ensure defined terms are used consistently.
• Verify that conclusions logically follow from the presented facts and legal principles.

Critical Questions:

• Does the model maintain a consistent interpretation of legal concepts throughout?
• Are there logical contradictions in how tests are applied?
• Did the model appropriately distinguish between majority judgment and dissent?
• Does the analysis acknowledge jurisdictional differences where relevant?

Practice Tip: Read the document in reverse order (conclusion first) to spot contradictions more easily.

3. Completeness Analysis

Key Action: Assess whether the model has addressed all necessary aspects of the task.

Process:

• Compare the output against a standard template or checklist for the specific task.
• Verify all necessary provisions are included.
• Identify any missing elements.
• Verify that relevant counterarguments have been considered.

Critical Questions:

• Has the model addressed all aspects of the tasks?
• Are there considerations that have been overlooked?
• Does the analysis account for counterarguments?
• Does the model fail to recognise when a case is distinguishable from the present facts?

Practice Tip: Develop issue-specific checklists for common tasks in your practice area

4. Relevance and Currency Check

Key Action: Confirm that the information is up-to-date and reflects current law.

Process:

• Confirm that all cited laws are applicable in the relevant jurisdiction
• Check the dates of cited cases and statutes
• Research whether any cases have been overruled or statutes amended
• Verify that recent significant developments in the area of law are incorporated

Critical Questions:

• Has the model mistakenly applied law from the wrong jurisdiction?
• Has the model cited superseded or overruled precedent?
• Does the analysis reflect the most current statutory language?
• Has the model accounted for any relevant legal changes after its knowledge cutoff date?
• Could current information affect the analysis?

Practice Tip: Include a timestamp on all AI-generated material to track currency.

5. Bias and Balance Assessment

Key Action: Evaluate whether the model has presented a balanced analysis.

Process:

• Identify whether the model has given appropriate weight to opposing arguments.
• Assess whether the model has considered diverse perspectives.

Critical Questions:

• Does the analysis present both sides of contested issues?
• Does the analysis acknowledge limitations in its conclusions?
• Has the model omitted relevant dissenting opinions or views?

Practice Tip: Always ask the model to generate arguments for both sides of a legal issue, even when only one perspective is needed for your final work product.

6. Legal Ethics Compliance

Key Action: Ensure the AI-generated content complies with professional responsibility obligations.

Process:

• Review the terms of service for an AI application
• Review for potential confidentiality issues
• Verify that representations about the law are accurate and not misleading

Critical Questions:

• Does the system and content maintain client confidentiality?
• Are there any misleading statements?
• Does the work product reflect adequate supervision of AI tools?

Practice Tip: Create an ethics compliance checklist specifically for AI-generated work products.

7. Remediation Process

When issues are identified in AI-generated content, follow this structured approach:

1. Document the Error: Record the specific issue, including what makes it incorrect or problematic
2. Research the Correct Information: Consult authoritative sources to determine the accuracy
3. Refine Your Prompt: Modify your initial instructions to help the model avoid similar errors
4. Test and Iterate: Generate a new version and verify if the issue has been resolved
5. Create a Knowledge Base: Maintain a record of common errors and effective remediation strategies

Practical Implementation

Consider implementing these techniques through a standardised workflow:

1. Initial Generation: Create AI-generated content using clear, detailed prompts.
2. Primary Review: Conduct a first-pass review focusing on apparent errors.
3. Structured Assessment: Apply the techniques above.
4. Targeted Refinement: Regenerate specific sections as needed.
5. Final Verification: Perform a comprehensive review of the final product.
6. Documentation: Record your verification process for professional responsibility purposes.

Strategies for improving outputs through prompt refinement

Effective use of generative AI requires evaluating outputs and refining prompts. As we have already discussed, the quality of AI-generated legal content depends on how the prompts are constructed.

Following the assessment techniques, this section explores practical techniques for refining prompts to enhance the quality of generative AI outputs.

Firstly, when optimising for results, approach prompt engineering systematically:

• (1) Diagnose the output: Consider the critical questions posed above.
• (2) Make targeted refinements: Apply strategies to address identified weaknesses (below).
• (3) Test refined prompts to evaluate improvement: Start a new chat with the model or engage in multi-step refinement (i.e., continue the same chat).
• (4) Review: Assess whether specific issues have been resolved.

1. Specify Legal Authority and Jurisdiction

If you find that outputs are generic or defaulting to American law, explicitly define the relevant legal framework to ensure outputs conform to the applicable jurisdiction.

For example:

Tips:

• Use constraints, for example, by specifying the jurisdiction (state/territory or Commonwealth).
• Augment an RAG process and provide the specific legislative provisions, regulations and cases when applicable.

2. Request Counterarguments

Direct the model to consider multiple perspectives to ensure a balanced analysis.

For example:

Tips:

• Request specific counterarguments to the main conclusions.
• Ask for alternative interpretations.
• Direct the model to identify different approaches.
• Request consideration of influential academic commentary from Australian sources.

3. Implement Constraint Prompting

Set specific boundaries to guide the model away from potential errors or irrelevant areas.

For example:

Tips:

• Use negative constraints to prevent unwanted content
• Define scope limitations clearly (what is in and out of bounds)
• Identify areas where speculation should be avoided

4. Apply Role-Based Perspectives

Direct the model to adopt specific legal roles to gain varied analytical perspectives.

For example:

Tips:

• Include both advocacy and adjudicative roles
• Request identification of different stakeholder interests
• Use roles to explore different perspectives

5. Use Iterative Refinement

Engage in multi-step refinement to progressively improve outputs.

Example process:

• (a) Generate an initial response.
• (b) Ask the model to identify potential weaknesses in its analysis.
• (c) Request a revised version addressing these weaknesses.
• (d) Pose follow-up questions on any remaining areas of concern.

Tips:

• Start with a broad analysis before narrowing the focus
• Request explicit identification of assumptions or weaknesses
• Ask for an application for specific hypothetical scenarios
• Build toward practical solutions or recommendations

7. Implement Document and Context Structuring

Provide clear guidance on how information should be organised and presented. See Starting to Prompt.

Implementation Tips:

• Specify document format and structure requirements.
• Set length parameters for different sections.
• Direct the model to use specific formatting (headings, bullet points).
• Request executive summaries or conclusions.
• Specify Australian spelling and terminology conventions.

Remember, while these strategies can significantly enhance AI outputs, they do not replace human legal expertise. Always apply your professional judgment and knowledge to verify and refine AI-generated content before using it in your work.