After six months of developing with the aid of AI tooling, I've gathered some insights and thoughts on the process, challenges, and potential future directions.

Important Metrics

We should be publishing comparison charts of context windows required (for a codebase or subsets of a codebase) and context windows available from the AI. This metric could provide valuable insights into the scalability and efficiency of AI-assisted coding.

Workflows

While the human is responsible for selecting changes to implement, the reality of the dynamic is that the AI will refactor potentially more often than the human in the loop would like. This frequent refactoring can lead to both benefits and challenges in the development process.

The Return of Waterfall?

Interestingly, waterfall methodology might be cool again. The skill will be to drive as much implementation as possible before testing. This approach could potentially leverage the AI's ability to generate large amounts of code quickly.

Communicating Status to AI

Perhaps what's needed is a method of communicating with the AI to provide context of the status of a diff. For example, indicating that a diff has been tested and works. Steps should be taken to preserve the inputs and outputs unless the user has explicitly asked for a refactor.

Iteration Tracking

Additionally, each diff could be tagged with the number of times the AI has processed the code. Each time the AI keeps a line, the associated integer gets incremented. This could provide insights into code stability and AI performance.

Challenges in the Development Process

A number of times I find myself rabbit-holing down the wrong path, generating some usable code on the way and then fixing the actual issue at the end of a loop. The results often have a mixture of useful and not useful code which needs to be picked over to remove redundant or suboptimal sections.

New Linting Tools

Perhaps there's a need for a new linting tool? More on this below. This may be the wrong paradigm for this capability and we need something entirely new, however it's worth exploring.

Future Developments

Of course, context windows will continue to grow. Maybe there will be a new framework composed to include everything required for a microservice into 50% of a context window, including both frontend and backend.

Maybe AI's will start to be trained on `.git` folders, such that they understand compressed code, and can read the history of a project (assuming it fits within the context window). This might facilitate debugging, and may give the AI some understanding of "production" vs "development".

AI-Assisted Coding Features

Here are some key features that could enhance AI-assisted coding:

• Function Intent Tracking
• Iteration History Logging
• AI Confidence Scoring
• Version Tracking
• AI Notes
• Function-Level Metadata
• Context window constraints in linters on file and project size

Example Code

Here's an example of how these features might look in practice:

            # Function Intent Tracking
            def process_image(image):
                """
                Initial Prompt: Create a function to process an image for facial recognition
                Iterations: 3
                Last Updated: 2024-10-06
                """
                # v1: Basic image loading
                image = load_image(image)  # Confidence: 0.95

                # v2: Added color normalization
                image = normalize_colors(image)  # Confidence: 0.88

                # v3: Implemented face detection
                faces = detect_faces(image)  # Confidence: 0.92

                return faces

            # Iteration History Logging
            """
            Iteration History:
            1. 2024-10-04: Initial implementation with basic image loading
               Prompt: "Create a function to load and process an image"
            2. 2024-10-05: Added color normalization for better recognition
               Prompt: "Improve image quality for facial recognition"
            3. 2024-10-06: Implemented face detection using a pre-trained model
               Prompt: "Add face detection to the image processing function"
            """

            # AI Confidence Scoring
            def train_model(X, y):
                """
                Initial Prompt: Implement a machine learning model for predictive analytics
                Iterations: 2
                Last Updated: 2024-10-06
                """
                # v1: Basic model creation and training
                model = create_model()  # Confidence: 0.97
                model.fit(X, y)  # Confidence: 0.99

                # v2: Added cross-validation
                scores = cross_validate(model, X, y)  # Confidence: 0.85
                print(f"Cross-validation scores: {scores}")  # AI Note: Consider using a logging framework instead of print

                return model

            # Example of a complex function with multiple iterations
            def preprocess_text(text):
                """
                Initial Prompt: Create a function to preprocess text for sentiment analysis
                Iterations: 4
                Last Updated: 2024-10-06
                """
                # v1: Basic lowercase conversion
                text = text.lower()  # Confidence: 0.99

                # v2: Added punctuation removal
                text = remove_punctuation(text)  # Confidence: 0.97

                # v3: Implemented tokenization
                tokens = tokenize(text)  # Confidence: 0.93

                # v4: Added stop word removal
                tokens = remove_stop_words(tokens)  # Confidence: 0.91

                # AI Note: Consider adding lemmatization or stemming in the next iteration

                return tokens

            """
            Iteration History:
            1. 2024-10-03: Initial implementation with lowercase conversion
               Prompt: "Create a basic text preprocessing function"
            2. 2024-10-04: Added punctuation removal for cleaner text
               Prompt: "Improve text cleaning by removing punctuation"
            3. 2024-10-05: Implemented tokenization for word-level analysis
               Prompt: "Add tokenization to the preprocessing function"
            4. 2024-10-06: Added stop word removal to focus on meaningful words
               Prompt: "Enhance preprocessing by removing common stop words"
            """
                

By incorporating these elements, the code becomes more than just a set of instructions. It becomes a living document that tells the story of its own development, facilitating better collaboration between AI and human developers. This approach allows for more transparent, traceable, and iterative development processes, which are crucial when working with AI-assisted coding.

A Linting function to populate a first draft prior to AI iteration.

                import ast
                import datetime
                import sys
                import argparse
                import random  # For generating placeholder confidence scores

                def ai_linter(code: str, prompt: str) -> str:
                    """
                    A specialized AI linter that adds initial metadata, standardized prompts for iteration history,
                    and per-line version numbers and confidence scores while preserving the original code structure and indentation.

                    :param code: The input Python code as a string
                    :param prompt: The initial prompt used to generate the code
                    :return: The linted code as a string
                    """
                    tree = ast.parse(code)
                    lines = code.split('\n')
                    linted_lines = []

                    # Add file-level metadata
                    today = datetime.date.today().strftime("%Y-%m-%d")
                    linted_lines.extend([
                        f'"""',
                        f'Initial Prompt: {prompt}',
                        f'Iterations: 1',
                        f'Last Updated: {today}',
                        f'"""',
                        ''
                    ])

                    for i, line in enumerate(lines):
                        stripped_line = line.strip()
                        if not stripped_line:  # Preserve empty lines without adding metadata
                            linted_lines.append(line)
                            continue

                        indent = len(line) - len(stripped_line)
                        space = ' ' * indent

                        # Generate a placeholder confidence score (replace this with actual AI confidence in a real scenario)
                        confidence = random.uniform(0.5, 0.65)

                        # Add version number and confidence score to each non-empty line
                        linted_line = f"{line} # v1 (Confidence: {confidence:.2f})"

                        for node in ast.walk(tree):
                            if hasattr(node, 'lineno') and node.lineno == i + 1:
                                if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
                                    # Add function/class level metadata
                                    linted_lines.extend([
                                        linted_line,
                                        f'{space}    """',
                                        f'{space}    Initial Prompt: {prompt}',
                                        f'{space}    Iterations: 1',
                                        f'{space}    Last Updated: {today}',
                                        f'{space}    """'
                                    ])
                                    break
                        else:
                            linted_lines.append(linted_line)

                    # Add iteration history
                    linted_lines.extend([
                        '',
                        '"""',
                        'Iteration History:',
                        f'1. {today}: Initial implementation',
                        f'   Prompt: "{prompt}"',
                        '"""'
                    ])

                    return '\n'.join(linted_lines)

                def main():
                    parser = argparse.ArgumentParser(description="AI Linter for Python files")
                    parser.add_argument("filename", help="The Python file to lint")
                    parser.add_argument("prompt", help="The initial prompt used to generate the code")
                    args = parser.parse_args()

                    try:
                        with open(args.filename, 'r') as file:
                            code = file.read()

                        linted_code = ai_linter(code, args.prompt)

                        output_filename = f"linted_{args.filename}"
                        with open(output_filename, 'w') as file:
                            file.write(linted_code)

                        print(f"Linted code has been written to {output_filename}")

                    except FileNotFoundError:
                        print(f"Error: The file {args.filename} was not found.")
                        sys.exit(1)
                    except Exception as e:
                        print(f"An error occurred: {str(e)}")
                        sys.exit(1)

                if __name__ == "__main__":
                    main()
                

This may be useful. I'll use it for a while and will report back. :)