You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the pipeline stops execution as soon as a binary match is achieved.
Proposed improvement
We can extend the pipeline to continue iterating after a match in order to improve code quality besides just matching it.
To achieve this, we need metrics to evaluate whether the generated code is improving over time. Possible approaches include:
Static analysis metrics such as cyclomatic complexity
Detecting known code smells (e.g. use of goto)
Asking an LLM to evaluate the code and provide a quality score based on the target codebase's standards
Once a match is found, the pipeline would enter a refinement phase where it attempts to improve code quality. The process stops when either:
A target quality score is reached, or
A maximum number of refinement attempts is exceeded
flowchart TD
A[Prompt Loader] --> |Load prompts from directory| B
B[Claude Runner]
C[Compiler]
D[Objdiff]
B --> |Generate C| C
C --> |Compilation Error → Retry| B
C --> |Compile to object file| D
D --> |Mismatch → Retry| B
D --> |Match found| E[Code Quality Refiner]
D --> |Max retries exceeded| F[Fail]
E --> J[Compiler]
J --> K[Objdiff]
J --> |Compilation Error → Retry| E
K --> |Mismatch → Retry| E
K --> |Max retries exceeded| G[Stop]
K --> |Target score achieved| H[Success]
Currently, the pipeline stops execution as soon as a binary match is achieved.
Proposed improvement
We can extend the pipeline to continue iterating after a match in order to improve code quality besides just matching it.
To achieve this, we need metrics to evaluate whether the generated code is improving over time. Possible approaches include:
goto)Once a match is found, the pipeline would enter a refinement phase where it attempts to improve code quality. The process stops when either:
flowchart TD A[Prompt Loader] --> |Load prompts from directory| B B[Claude Runner] C[Compiler] D[Objdiff] B --> |Generate C| C C --> |Compilation Error → Retry| B C --> |Compile to object file| D D --> |Mismatch → Retry| B D --> |Match found| E[Code Quality Refiner] D --> |Max retries exceeded| F[Fail] E --> J[Compiler] J --> K[Objdiff] J --> |Compilation Error → Retry| E K --> |Mismatch → Retry| E K --> |Max retries exceeded| G[Stop] K --> |Target score achieved| H[Success]