WELCOME TO LIGHTHOUSE!

Ahoy! Welcome to the Lighthouse!

The Lighthouse project encompasses four different specific workflows: Summarization (CYU), Text Composition (TA/TC), Visual Content Generation (VCG), and DE (Device Expert).   While they share some points in common, each of these workflows should be considered separately and studied independently. Make sure to visit the Guidelines links banner at the top of the page when working on a specific workflow!

IMPORTANT ⇒ Lighthouse Production Guidelines and Updates

IMPORTANT:

• For CYU and TC, it is important that you also read the General Guidelines, as these explain the general approach. However, please note that it is essential that you go over the guidelines for the workflows you are working on (CYU, TC, VCG and DE).
  
• Simplify! Please remember that you can ONLY work on the workflows for which you have passed the certification. Focus your efforts on reading the information and guidelines relative to the workflows you are actually working on.
  

Important: general order in which the certifications should be taken is:

1. LE (Localization)
   • 1 hour
   • Confirms that you’re a native speaker
   • Mandatory for participating in any of 4 sub-projects/workflows
2. CYU and TA/TC (Summarization and Text Composition)
   • 1.5 hours each
   • Mandatory to participate in production for CYU or TA/TC workflow
3. VCG (Visual Content Generation)
   • 1 hour
   • Mandatory to participate in production for VCG workflow
4. DE (Device Expert)
   • 1 hour
   • Mandatory to participate in production for DE workflow

Important notes:

• For each certification, the client is providing training and preparation material
• For all certifications, you have 2 attempts.
• For CYU, TA/TC, VCG and DE certs – there are practice tasks before moving on to cert

Please find below an overview of each workflow:

LE Certification is the main pre-requisite for the project and should be done first, since they cannot qualify for production of any of the sub-projects if they don’t pass it. If they fail it twice, they are automatically disqualified from the project.
LE Certification ⇒ LE Certification

The task involves evaluating summarization responses generated by a virtual assistant. The process consists of examining instructions and original input text (which could be emails, chat messages, or other forms of text) and then evaluating the assistant's summary based on specific criteria.
CYU Certification ⇒ CYU Certification

The Text Composition tasks involve generating or evaluating various text types, such as replies, rewrites, tone adjustments, and text-to-list or table transformations. These tasks focus on ensuring the text follows instructions, remains grounded in the input, is comprehensive, and maintains proper tone, style, and grammar. The TC workflow involves several task types (e.g., Tone Adjust, List Transform, Key Points Transform, etc.), each with individual guidelines. Please make sure you are following the guidelines for the appropriate workflow!
TA/TC Certification ⇒ TA/TC Certification

This task involves evaluating the quality of machine-generated images based on prompts that may include text, sketches, or emojis. Evaluators review each output image across key dimensions: Structural Integrity and Input/Output Alignment. Structural Integrity ensures that images depict anatomically and visually correct subjects (e.g., humans, animals, objects) without distortions, artifacts, or missing features. Input/Output Alignment assesses how well the image reflects the prompt's specified elements, mood, and spatial layout.
Visit the General Guidelines
Visit the VCG guidelines :
VCG Image Evaluation Guidelines
Emoji Evaluation Design
Handwriting Synthesis and Refinement SBS Guidelines
Image-to-Image Eval
VCG - Overview November 14 2024

The goal of this sub-project is to evaluate the quality of a machine-generated answer in response to a user's question about their product(s). The answer should deliver the relevant information as clearly and concisely as possible.
DE Certification ⇒ DE Certification *page is under development…

Aside from the General Guidelines, we have a Safety Evaluation Harmfulness Guidelines , which outline a framework for assessing harmful content in user prompts and responses, categorizing it into main harm types. Each includes specific categories, covering a range of issues (hate speech, violence, etc.).

The overarching goal is to protect user safety, privacy, and mental well-being while ensuring content is accurate, balanced, and compliant with ethical standards.

As all the workflows involve taking into account potential harm or safety concerns in the outputs, please make sure to go over these guidelines and consider them during your work.

The client has provided a set of helpful translations for the terms relevant to the project to help ensure a consistent understanding of their platform and guidelines across multiple languages. You will be able to find the general terms found on the platforms and guidelines in your own language. It’s a handy reference for you to better understand the project, in case the English terms are not clear enough.

Check it out here!

Note: There are a few terms in the French section that are missing translations—we’re still waiting for the client’s confirmation on those. Keep this in mind if you’re working on French tasks, and feel free to flag any issues or questions.

All of these projects share an element in common: Evaluating the output of an AI Agent (be it text or images) based on defined criteria, and then ranking the outputs generated for similar inputs on a scale based on some of these defined dimensions.

This is generally known as Preference Ranking.

Particularly for CYU and TC, please read the General Guidelines in the General Guidelines section to get a general notion of Preference Ranking projects, but do not forget to visit and study the specific guidelines for each workflow (including VCG!) to get a detailed description of the steps and requirements involved.

HAPPY RANKING!

NEXT ⇒ TA/TC Certification