User Tools
karl
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
karl [2025/04/22 15:15] sergio |
karl [2025/06/12 07:36] (current) sergio |
||
|---|---|---|---|
| Line 9: | Line 9: | ||
| This project will consist of two different workflows or tasks, see below: \\ | This project will consist of two different workflows or tasks, see below: \\ | ||
| + | |||
| + | * **[[https://nm.oho2.com/s/T7NCcKqMpmmbXc3|Centific - Karl - 3P Prompt Rewrite 2]]**\\ | ||
| + | * **[[https://nm.oho2.com/s/gdXKPwBBrFcMTkN|JSON content instruction]]**\\ | ||
| + | * **[[https://nm.oho2.com/s/NyfMZKpgacNwM9g|Uber_Project_Karl]]**\\ | ||
| + | |||
| + | |||
| + | |||
| **Quality Grading Task (SxS)**: In this workflow, we will evaluate translation quality automatically generated. Your task will be to choose from a range of 0 to 6 the quality level for each of the translations that appear as options based on 2 error categories: accuracy and fluency. \\ | **Quality Grading Task (SxS)**: In this workflow, we will evaluate translation quality automatically generated. Your task will be to choose from a range of 0 to 6 the quality level for each of the translations that appear as options based on 2 error categories: accuracy and fluency. \\ | ||
| Line 52: | Line 59: | ||
| **Prompt Evaluation and Rewrite**: This task consists of reviewing a prompt based on 3 quality dimensions (self-contained, naturalness and clear intent): if at least one of these parameters are not met, then your task is to also rewrite the prompt in order to meet these 3 parameters. \\ | **Prompt Evaluation and Rewrite**: This task consists of reviewing a prompt based on 3 quality dimensions (self-contained, naturalness and clear intent): if at least one of these parameters are not met, then your task is to also rewrite the prompt in order to meet these 3 parameters. \\ | ||
| - | //All information is here:// | + | //All information is here:// \\ |
| - | [[http://example.com|External Link]]\\ | + | * [[https://nm.oho2.com/s/TL3CiYsy2q3i6tS|Prompt Eval Rewrite_Guidelines_v.1]]\\ |
| + | * [[https://nm.oho2.com/s/ZKWJcN8GCME58wq|IMPORTANT-rework_Prompt Eval and rewrite]]\\ | ||
| + | * [[https://nm.oho2.com/s/jkPsFK4fEH3eBFw|Promp_Evaluate_Rewrite_training]]\\ | ||
| - | **Prompt Translation Task**: In this project you will have to translate a given prompt and ensure the translated version meets the needed requirements stated in the guidelines. You do not need to review the prompt in terms of correctness, accuracy, etc. as in other projects. All information is here. [[http://example.com|External Link]]\\ | + | **Prompt Translation Task**: In this project you will have to translate a given prompt and ensure the translated version meets the needed requirements stated in the guidelines. You do not need to review the prompt in terms of correctness, accuracy, etc. as in other projects.\\ |
| + | //All information is here://\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/RCTq8ScmWDSmQEw|Review_AI Style guide for Human Evaluations ]]\\ | ||
| + | * [[https://nm.oho2.com/s/SG4PaYMNRbXrqjG|Translation_AI Style guide for Human Evaluations]]\\ | ||
| + | * [[https://nm.oho2.com/s/jop9fQdW6CePdGt|CrC Translation Instructions_V2]]\\ | ||
| + | * [[https://nm.oho2.com/s/fgTYEkSe6tN222o|Training session for Karl_Loc_Project-20240515_CN]]\\ | ||
| + | * [[https://nm.oho2.com/s/ixYxZYWjoML2Htr|Training session for Karl_Loc_Project-20240515_EN 2]]\\ | ||
| + | |||
| + | **i18N question data Post-editing**: In this task, according to the translation output, you will need to add a level of quality according to the presence of issues in the localized output, and post-edit the final result to make it error-free. \\ | ||
| + | //All instructions are here//:\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/qF9m9EgY7yQDGkL|i18N Question Data Post-editing Instruction]]\\ | ||
| + | |||
| + | **Source Text Translation with Glossary Hints**: In this project you will have to translate a given text to the required target language with proper grammar, punctuation, spelling and maintaining the source meaning intact by using a given glossary term.\\ | ||
| + | //All information is here://\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/tqotXbgTpkfGeWN|Source Text Translation with Glossary Hints_v1.0]]\\ | ||
| + | |||
| + | **Fluency and Adequacy Rating**: The goal of this task is to evaluate the MTed conversational data from a human standpoint assessment. You will need to read the original text and translated text to determine the quality based on the adequacy and fluency, as main parameters. \\ | ||
| + | //Detailed instructions are here://\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/JC677kFEw8iCGPz|Fluency and Adequacy Rater Instructions]]\\ | ||
| + | |||
| + | **Transcribe Audio - Food Orders**: For this project, you will hear an audio of customers ordering at a drive through for fast food. Your task will be to rate the audio transcript for accuracy by using a specific guideline here provided:\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/7Y4fmBdapmHQj6B|Transcribed Audio - Food Orders v2]]\\ | ||
| + | * [[https://nm.oho2.com/s/H3jMASb54aQwNaA|Karl - Audio Transcription - ES - Training Meeting-20240806_170640-Meeting Recording]]\\ | ||
| - | **i18N question data Post-editing**: In this task, according to the translation output, you will need to add a level of quality according to the presence of issues in the localized output, and post-edit the final result to make it error-free. All instructions are here.[[http://example.com|External Link]]\\ | + | **Prompt and Response Generation - Indian Languages**: In this project, you will need to create a prompt and response (a pair) in your specific dialect for Indic languages. These prompts should be diverse, of different levels of severity. \\ |
| + | //See info here://\\ | ||
| - | **Source Text Translation with Glossary Hints**: In this project you will have to translate a given text to the required target language with proper grammar, punctuation, spelling and maintaining the source meaning intact by using a given glossary term. All information is here. [[http://example.com|External Link]]\\ | + | * [[https://nm.oho2.com/s/33yeJQqqtSqFpZz|Evaluators - Prompt & Response Generation Indic -Task ]]\\ |
| + | * [[https://nm.oho2.com/s/2mZZ3myQbeoKeQ2|Reviewers- Prompt & Response Generation Indic -Task]]\\ | ||
| + | * [[https://nm.oho2.com/s/3SgGBXSqQ2SJKfw|Prompt & Response Generation Task Training ]]\\ | ||
| - | **Fluency and Adequacy Rating**: The goal of this task is to evaluate the MTed conversational data from a human standpoint assessment. You will need to read the original text and translated text to determine the quality based on the adequacy and fluency, as main parameters. Detailed instructions are here. [[http://example.com|External Link]]\\ | ||
| - | **Transcribe Audio - Food Orders**: For this project, you will hear an audio of customers ordering at a drive through for fast food. Your task will be to rate the audio transcript for accuracy by using a specific guideline here provided.[[http://example.com|External Link]]\\ | + | **Severity and Safety Labeling**: You will need to evaluate both prompt and responses based on how harmful the content is. You will have a definition for each safety attribute (hate speech, harassment, dangerous content, sexually explicit) for both prompt and response, and also evaluate the quality (high and low) for each pair (both prompt and response overall assessment), according to a criteria set for this task.\\ |
| + | //All the detailed guidelines are found here://\\ | ||
| - | **Prompt and Response Generation - Indian Languages**: In this project, you will need to create a prompt and response (a pair) in your specific dialect for Indic languages. These prompts should be diverse, of different levels of severity. See info here.[[http://example.com|External Link]]\\ | + | * [[https://nm.oho2.com/s/GFEEANrd7PcXAYS|Severity and Safety Labeling Guidelines]]\\ |
| - | **Severity and Safety Labeling**: You will need to evaluate both prompt and responses based on how harmful the content is. You will have a definition for each safety attribute (hate speech, harassment, dangerous content, sexually explicit) for both prompt and response, and also evaluate the quality (high and low) for each pair (both prompt and response overall assessment), according to a criteria set for this task. All the detailed guidelines are found here. [[http://example.com|External Link]]\\ | ||
| **Severity and Safety Labeling**: The evaluation encompasses three tasks for the prompt and for the response. The response will be evaluated with respect to the prompts. The annotations are based on below points:\\ | **Severity and Safety Labeling**: The evaluation encompasses three tasks for the prompt and for the response. The response will be evaluated with respect to the prompts. The annotations are based on below points:\\ | ||
| Line 77: | Line 115: | ||
| **Manual SxS Human Evaluations**: In this task, you will see a user prompt and two AI-generated responses [or responses from 2 different AI models]. You will assess each response itself along seven dimensions, and at the end, you will select which response you think is better. You will provide an explanation as to why you think it is better. \\ | **Manual SxS Human Evaluations**: In this task, you will see a user prompt and two AI-generated responses [or responses from 2 different AI models]. You will assess each response itself along seven dimensions, and at the end, you will select which response you think is better. You will provide an explanation as to why you think it is better. \\ | ||
| - | The dimensions are: Safety/Harmlessness, Writing Style, Verbosity, Instruction Following, Truthfulness, Overall Quality. Then, you will need to rewrite your response based on the Style guide provided in the instructions here[[http://example.com|External Link]].\\ | + | |
| + | The dimensions are: Safety/Harmlessness, Writing Style, Verbosity, Instruction Following, Truthfulness, Overall Quality. Then, you will need to rewrite your response based on the Style guide provided in the instructions here:\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/PqmaqeRtAgJTnJ6|i18n Manual SxS Human Eval]].\\ | ||
| + | * [[https://nm.oho2.com/s/PqmaqeRtAgJTnJ6|Manual SxS Human Evals Training]]\\ | ||
| **Cultural Relevance Benchmarking Rubric**: In this task, you will see a user prompt and an AI-generated response. You are required to evaluate the provided prompt and its corresponding response keeping the Indian culture in mind. In other words, we need your help in identifying the “culturally (Indian) relevant” responses. | **Cultural Relevance Benchmarking Rubric**: In this task, you will see a user prompt and an AI-generated response. You are required to evaluate the provided prompt and its corresponding response keeping the Indian culture in mind. In other words, we need your help in identifying the “culturally (Indian) relevant” responses. | ||
| - | In this task , both prompt and response are expected to be in Hindi language only. Some of the prompts or responses may contain English words along with Hindi. More detailed guidelines are here.[[http://example.com|External Link]]\\ | + | In this task , both prompt and response are expected to be in Hindi language only. Some of the prompts or responses may contain English words along with Hindi. \\ |
| + | //More detailed guidelines are here://\\ | ||
| + | |||
| + | * [[https://nm.oho2.com/s/73oK47tXniRZo6E|Guidelines for HI Cultural Evaluation]]\\ | ||
| + | * [[https://nm.oho2.com/s/pCLtxGiESw9j7HA|Instructions - Cultural Benchmarking Rubric - Hindi]]\\ | ||
| + | * [[https://nm.oho2.com/s/NKWAXCnNiA9QxNi|Cultural Relevance Benchmarking Rubric - AI Data v2]]\\ | ||
| + | * [[https://nm.oho2.com/s/k9ZJCNnJymkzqsK|Cultural Relevance Benchmarking Rubric - Phase 2 - HI]]\\ | ||
| + | * [[https://nm.oho2.com/s/zndfMtXxN7cQoQH|Training India Cultural Evals]]\\ | ||
| + | * [[https://nm.oho2.com/s/MienH725rfagXdN|Rating Criteria Detailed Explanation]]\\ | ||
| + | |||
| === Localization === | === Localization === | ||
| - | **Human Translation and Review projects**: This is a localization task that will have to work on Phrase as a TMS. There are no specific language style guides, but general linguistic considerations to be applied in your translations explained in this document[[http://example.com|External Link]]. The relevant folder with all the needed info can be found here.[[http://example.com|External Link]] \\ | + | **Human Translation and Review projects**: This is a localization task that will have to work on Phrase as a TMS. There are no specific language style guides, but general linguistic considerations to be applied in your translations explained here:\\ |
| + | |||
| + | * [[https://nm.oho2.com/s/ttWMnMqRMQwoRjn|Karl_Human Translation and Review Guideline.v.1]]\\ | ||
| + | * [[https://nm.oho2.com/s/poAB77FrY4PjX8Y|Phrase_General_linguists_guide]]\\ | ||
| + | * [[https://nm.oho2.com/s/SPpLACxZk9aqCaF|Phrase_Quick manual]]\\ | ||
| + | * [[https://nm.oho2.com/s/25wRrf8F9WrsRjX|CENTIFICNoMT_Scorecard_Template_1.1_Test_Updated]]\\ | ||
| + | * [[https://nm.oho2.com/s/HrDQCqxobtYNMTa|Translation and Review HO]]\\ | ||
| + | * [[https://nm.oho2.com/s/EBHHfyriTAz75qG|Arabic UI glossary]]\\ | ||
| + | * [[https://nm.oho2.com/s/xF3BLTncqSDkq8b|Taptap Send Localization Style Guide Arabic]]\\ | ||
| + | * [[https://nm.oho2.com/s/bL3MZn6qLPHTyMy|TapTap_AR_glossary checklist]]\\ | ||
| + | * [[https://nm.oho2.com/s/NmAs62RY764dSDF|Arabic app - Translation Brief]] | ||
| Line 91: | Line 153: | ||
| We will be working in a new tool hosted, and developed by the client, called **Data Compute**. To be able to use this tool, first, you will need to create your own account. As always, security is key, so you might be requested to provide your email, your telephone number and some personal details who are not to be revealed anywhere, but just with the purpose of creating your client’s dedicated account. \\ | We will be working in a new tool hosted, and developed by the client, called **Data Compute**. To be able to use this tool, first, you will need to create your own account. As always, security is key, so you might be requested to provide your email, your telephone number and some personal details who are not to be revealed anywhere, but just with the purpose of creating your client’s dedicated account. \\ | ||
| That account will be valid for all the duration of the project, and you will be receiving the details by our team once all the information is passed on to you.\\ | That account will be valid for all the duration of the project, and you will be receiving the details by our team once all the information is passed on to you.\\ | ||
| - | The step-by-step instructions along with an FAQ document is available at all times here[[http://example.com|External Link]]. These is also a separate folder with the needed documentation for further reference here.[[http://example.com|External Link]]\\ | + | The step-by-step instructions along with an FAQ document is available at all times here:\\ |
| + | |||
| + | * [[https://nm.oho2.com/s/QRxN6L9mHkdmTaE|Detailed Step by Step Guide To Start Working & FAQ]]\\ | ||
| + | * [[https://nm.oho2.com/s/apNcyXkfPZgSwEY|Detailed Instructions for the setup and accessing the tasks]]\\ | ||
| + | * [[https://nm.oho2.com/s/Qfk9MWm49p75RA3|Uber Tool User Creation account]]\\ | ||
| + | * [[https://nm.oho2.com/s/FAaJYPmCzfiWjbX|Account_Issues_Workarounds]]\\ | ||
| === Phrase as Translation Management System - Localization Projects === | === Phrase as Translation Management System - Localization Projects === | ||
| - | The TMS for Karl projects is Phrase (Memsource). Details on how to perform your translation or review, and the mandatory checklists can be found here [[http://example.com|External Link]] \\ | + | The TMS for Karl projects is Phrase (Memsource). Details on how to perform your translation or review, and the mandatory checklists can be found here:\\ |
| + | [[https://nm.oho2.com/s/poAB77FrY4PjX8Ym|Phrase_General_linguists_guide]] \\ | ||
| + | **Rubric Review** | ||
| + | The goal of this task is to review each individual rubric property for a prompt. Assume you are a teacher responsible for scoring student responses and validate, correct, or remove the given rubrics.\\ | ||
| + | More detailed guidelines are here:\\ | ||
| - | * **[[https://nm.oho2.com/s/T7NCcKqMpmmbXc3|CrC Translation Instructions]]**\\ | + | * [[https://nm.oho2.com/s/jjs2Y4H6pWNN4xZ|Rubric Review]]\\ |
| - | * **[[https://nm.oho2.com/s/T7NCcKqMpmmbXc3|Centific - Karl - 3P Prompt Rewrite 2]]**\\ | + | * [[https://nm.oho2.com/s/xAE9cXQ5oWdqnRH|Guidelines for Rubric Review]]\\ |
| - | * **[[https://nm.oho2.com/s/tjHHW8HALpLj2B9|i18N Question Data Post-editing Instruction]]**\\ | + | * [[https://nm.oho2.com/s/eLNRSBxKPkfKfyR|Rubric Review Training]]\\ |
| - | * **[[https://nm.oho2.com/s/gdXKPwBBrFcMTkN|JSON content instruction]]**\\ | + | * [[https://nm.oho2.com/s/9nZDE5tigkBddNq|RUBRIC REVIEW - DO’S AND DON'TS]] |
| - | * **[[https://nm.oho2.com/s/R8B8BNXtqQpJfKs|Karl_Human Translation and Review Guideline.v.1 ]]**\\ | + | |
| - | * **[[https://nm.oho2.com/s/68ifaJZXoGEit46|Training session for Karl_Loc_Project]]**\\ | + | |
| - | * **[[https://nm.oho2.com/s/387MbnizT6pgLCy|Translation AI Style guide for Human Evaluations ]]**\\ | + | |
| - | * **[[https://nm.oho2.com/s/NyfMZKpgacNwM9g|Uber_Project_Karl]]**\\ | + | **Payment information** : [[:payment_terms_trs|Payment information for translators]]\\ |
| - | * **Payment information** : [[:payment_terms_trs|Payment information for translators]]\\ | + | |
karl.1745334918.txt.gz · Last modified: 2025/04/22 15:15 by sergio
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
