Question Regarding Prompt Format for aiXcoder-7B-v2 on CrossCodeEval Benchmark

[ZhenruiLi]

Hi, could you clarify what prompt format aiXcoder-7B-v2 uses for the CrossCodeEval benchmark?

Based on the implementation in prompt_aixcoder_colt.py script from this repository, we used the following prompt format. However, we weren’t able to reproduce the expected performance on CrossCodeEval.

Here's an example of the prompt format we utilized:

<s><!--the file path is: README.md-->
<!--the code file is written by Markdown-->
# Task Description:
Your task is to complete an incomplete Code file. The input includes in-file context and cross-file context, as described below:
# Explanation of Inputs:
1. In-file Context:
  - Prefix (Context Before): The code that appears before the missing section.
  - Suffix (Context Below): The code that follows the missing section.
  Your task is to predict the code that should be placed between the prefix and suffix.
2. Cross-file Context:
  - Similar Code Snippets: Code snippets from the current project that resemble the desired implementation.

# Input Structure:
  - Cross-file Context:
    - Similar Code Snippets
  - In-file Context:
    - Prefix
    - Suffix

</s>// Similar Code Snippets:
<s>// the file path is: src/main/java/io/blamer/hub/pg/Validated.java
// the code file is written by Java
              "Chat ID %s already exists"
                .formatted(id)
            )
          )
      ).switchIfEmpty(this.chats.add(chat));
  }
}</s><s>// the file path is: src/main/java/io/blamer/hub/pg/PgChat.java
// the code file is written by Java
public class PgChat implements Chat {
  /**
   * Chat ID.
   */
  private final long id;
  @Override
  public long id() {
    return this.id;
  }
}</s><s>// the file path is: src/main/java/io/blamer/hub/pg/PgToken.java
// the code file is written by Java
  private final long chat;
  @Override
  public Long id() {
    return this.id;
  }
  @Override
  public String value() {
    return this.value;
  }
  @Override</s><s>// the file path is: src/main/java/io/blamer/hub/pg/Validated.java
// the code file is written by Java
    return this.db.sql("SELECT c.id AS id FROM chat c WHERE c.id = :chat")
      .bind("chat", chat.id())
      .fetch()
      .one()
      .flatMap(rows ->
        Mono.just((Long) rows.get("id")))
      .flatMap(
        (Function<Long, Mono<Void>>) id ->
          Mono.error(
            new ChatAlreadyExists(</s><s>// the file path is: src/main/java/io/blamer/hub/pg/PgTokens.java
// the code file is written by Java
  public Mono<Void> add(final Token token) {
    return this.db.sql(
        "INSERT INTO token (token, alias, chat)"
          + "VALUES (:token, :alias, :chat)"
      )
      .bind("token", token.value())
      .bind("alias", token.alias())
      .bind("chat", token.chat().id())
      .fetch()
      .first()</s>// Fill-in-the-Middle Task

<s>▁<AIX-SPAN-PRE>▁<AIX-SPAN-POST>
  }
}
▁<AIX-SPAN-MIDDLE>// the file path is: src/main/java/io/blamer/hub/pg/PgChats.java
// the code file is written by Java
package io.blamer.hub.pg;

import io.blamer.hub.model.Chat;
import io.blamer.hub.model.Chats;
import org.springframework.r2dbc.core.DatabaseClient;
import org.springframework.stereotype.Component;
import reactor.core.publisher.Mono;

/**
 * Chats in PostgreSQL.
 *
 * @author Aliaksei Bialiauski (abialiauski.dev@gmail.com)
 * @since 0.0.0
 */
@Component
public class PgChats implements Chats {

  /**
   * Database Client.
   */
  private final DatabaseClient db;

  /**
   * Ctor.
   *
   * @param dbc Database Client.
   */
  public PgChats(final DatabaseClient dbc) {
    this.db = dbc;
  }

  @Override
  public Mono<Void> add(final Chat chat) {
    return this.db.sql(
        "INSERT INTO chat (id) VALUES (:id)"
      )      // The completed code:

We simply removed the Global Declarations part from the prompt to align with the CrossCodeEval benchmark and kept everything else unchanged. Could that be the reason for the performance gap? Or is there something else we may have missed?