v2.3.0

Tutorial

Content

How to install and run Markyannotation software
Administrators manual
Annotators' manual
1. Annotation functionalities
2. Recurring doubts in annotation

How to install and run Marky

The following dependencies are required to install this annotation tool

A database engine such as MySQL, Oracle, etc.
Apache Web server, with PHP 5.5 installed and the module mod_rewrite enabled

Then you should follow the steps below.

Donwload Marky
The folder Marky should be placed in the directory of apache /www.
Add permissions readable, writable and executable to Marky folder.
Edit app/Config/core.php and change Security.salt and Security.cipherSeed. Alter the code according to your preference.

Edit app/Config/database.php and set the user account to be used in accessing the database. For example, a MySQL account with user "server" and the password "server" would look like:

 public $default = array ( 
 'datasource' => 'Database / Mysql',
 'persistent' => false, 
 'host' => 'localhost', 
 'login' => 'server', 
 'password' => 'server', 
 'database' => 'marky', 
 'prefix' =>'' );

You need to modify the file .htcacess in Marky directory as follows:

 # Options Indexes FollowSymLinks MultiViews 
 # AllowOverride All # HERE PUT None, change to All 
 # Order allow, deny # allow from all

Then run the script marky.sql.
Finally, you have to access the Marky's web and create the first administration account.

This annotation tool supports two models of resource elimination.

If you have the database configured with delete cascade, you should set the variable deleteCascade in file app/Config/core.php to false (default option):
```
 Configure::write('deleteCascade', false);
                                
```
if you want to make Marky responsible for removing all database resources , you should set the variable deleteCascade on app/Config/core.php to true
```
 Configure::write('deleteCascade', true);
                                
```

How to configure the email component?

In order to that tool be able to send results to your email, or to reset your password. you should edit the file app/Config/email.php.and specify an email account or an SMTP server.For example:

 class EmailConfig {
 public $gmail = array(
 'host' => 'smtp.gmail.com',
 'port' =>,
 'username' => 'my@gmail.com',
 'password' => 'secret',
 'transport' => 'Smtp',
 'tls' => true
 );
 }

and you should include your email profile in app/Config/core.php

 Configure::write('emailProfile', 'gmail');

So the emails are sent from the gmail account named to my. More information here.

Also, you should change the maximum execution time of long duration scripts

 Configure::write('scriptTimeLimit', 3600);

Administrator Manual

Project Life-cycle

A project accounts for the following main components set of documents or corpus, species or concepts of interest, annotations, and users (administrator and annotators). The project administrator and the team of annotators have one shared goal: to carry out the work adequately to meet the project's objectives. At initiation, the annotation goal of the project is established and the team is defined. The documents to be annotated are automatically retrieved from an online source or uploaded by the administrator. The concepts of interest, in particular the different types of concepts and their association, are identified manually and their semantics is formalised in a set of annotation guidelines.

This formalisation typically occurs after the preliminary meetings with annotators, and is meant to guide the annotation process, help annotators decide on (or disambiguate) textual mentions, and leverage annotators' domain expertise. Often, the annotation project involves several annotators and is conducted in multiple rounds to guarantee the quality of the final annotations. This annotation tool keeps track of the work done by each annotator at every round. Annotation rounds may prompt unanticipated issues, which may lead to changes in annotation guidelines, and even in the set of annotation types. Therefore, each round of annotation has associated its own set of guidelines and concept types.

Posts

Example of flexibility on the cover of Marky

Marky gives administrators the option to create a custom home page for the application. Marky's cover is intended to be occupied by the news that administrators want to communicate to their annotators. To create a new post, or editing an existing one, the administration should access the menu area about posts.

How to specify which documents are to be annotated?

Retrieve documents from PubMed
To retrieve documents from Pubmed, you should indicate the Pubmed identifiers(PMIDS) of the intended documents, one PMID per line
You can import documents from PubMed only if they are also in PMC
Upload local documents
This functionality allows you to upload different types of documents, such as.html, .txt, and .htm from local sites
It is there also possible to put the URL of the HTML page source, in order to repair links like images, hyperlinks, etc.
For example, to upload a document from Wikipedia, you should specify the URL www.en.wikipedia.org and then have all associated links redirected to this page.
Create my own documents
This feature allows you to create a document with all the possibilities of html (images, links, etc or even youtube videos!). The interface is similar to creation of blogs or typing a document such as Microsoft Office Word.
All documents can be edited until the annotation rounds start

How to create a user profile?

There are two profiles available:

Create an annotator profile
Create a project administrator profile
Only the administrators can create annotation profiles

How to create an annotation project?

To create a project, you should click on the menu tab "New Project". and then introduce a name and a description for the new project.

The next step is to select the documents that are part of the project and the annotators that participate in it. Finally, click on save.

How to create a type or class of annotation?

You can create annotation types directly from Marky's main menu or from the view of project.

You have the possibility to add the new type to all project's rounds or just a subset.

Also you are asked to associate a color, which will denote the type in the document annotation view.

It is advisable not to choose dark colors, for readability sake. Alternatively you can even create a transparency effect on the color. Transparency can be adjusted by clicking on the checkbox A(alpha).

Finally, you should add a description to the type, to help annotators understand its semantics.

Example of creating annotation class or type interface

How to import types from other projects?

This annotation tool allows yo share annotation types among projects. For this, you should go to "View Project>Import types". From the list of all existing types, you may choose the types to be "shared".

How to associate notes to annotation types?

The association of questions or notes to annotation types aims to help annotators understand some of the targeted semantics, to create a question for one type or class of annotation, you should go to "View type", select "New question", and enter the text of the new question.

How to create a round?

After instantiating the project (documents,annotation types and annotators), you should create rounds of annotation.

The option to create a round is only be available at the project's view. Have the option "New Round". Here, enter a title, a date of completion and a description.

From this point forward, annotators may start annotating the documents.

Example of the interface where is the option to create a question for a Type.

How to create a version of a round?

First you should change the end date of the Round to a date earlier than the current. Then, go to Project>view>Round>create a new version of round and choose a new name, the original round and all annotators that will participate in the new version of this round.

This process may take some time, meanwhile you can perform other operations and monitor the progress in the project's view.

How to delete a round?

The procedure is very simple, to remove a Round just click the option.

However, It should be noted that you cannot delete a Round while another round is being copied and if it still open.

How to inspect annotations and assess agreement?

By monitoring annotation changes, the project administrator may supervise the compliance of annotators with annotation guidelines and thus, adjust these guidelines to enhance comprehensibility and alert annotators about incorrect or dubious curation patterns. Notably, it is highly unlikely that two annotators tag the very same text fragments, or completely agree on text-concept associations. Mostly, variability arises from differences in domain expertise, annotation skills, and interpretation of annotation criteria. From the view of the project, it is possible to run annotation agreement assessments:

among rounds
among annotators
among types
F-score for two annotators
F-score for two rounds
Load table from file

All options for agreement statistics have the possibility to specify a margin of error. Below is an example of calculation with a margin of error of 2 characters in matching annotations:

Annotation 1: A gene is a molecular unit of heredity of a living organism.

Annotation 2: A gene is a molecular unit of heredity of a living organism.

Annotation 3: A gene is a molecular unit of heredity of a living organism.

.Ejxample of a project view, where you can access the concordance tables.

View annotator statistics

This annotation tool allows you to view statistics for each annotator in a project. To do this, you go to visit the "project view" and chosen the option "Show statistics".

How to see all annotations in a round?

To see all annotated documents in one Round, you need to go "Project view>Round view>View Documents Annotated". Below you can see a snapshot of this operation.

Load statistics

You can view past statistic that have been downloaded or have been sent to your email (send me an email option from any of the statistics). To do so just go to the corresponding project and click the option view project > menu > get agreement tables > load table from file

Calculate agreement

The improvement in the rates of annotation agreement is quantified using the F-score, a common metric in IAA evaluations presented below:

F-score=2*(precision*recall)/(precision+recall)

Precision=(number of identical entities in set A and set B )/(number of entities in set A)

Recall=(number of identical entities in set A and set B)/(number of entities in set B)

such that Set A refers to the set of annotations produced by annotator A and set B refers to the set of annotations produced by annotator B, and recall(set A, set B)=precision(set B, set A)

Agreement among types

This option allows you to observe annotation agreement in more detail, (among two annotators and among two rounds). You can see hits and mismatches ("annotation by only one user") among the namely different types.

Clicking on the hits buttons, you can get information about the annotations and answers provided by the annotators.

Agreement among annotators

Annotator statistics per round describe the volume of annotations achieved by the annotator for the different types of concepts in analysis in different rounds. The steps are similar to the previous option, but in this case you can see the hits between one or more rounds for the same user.

Agreement across rounds

This option aims to obtaining a quality measure among the various rounds of one user throughout the project. In this calculation, you should specify a margin of characters between init and end of annotation.

To list the concordances, you should select the list of rounds that you want to compare. Then, a screen where you can see the number of "hits" between rounds (one hit is a fragment of text that has been annotated by one or more user for the same type).

By clicking on the obtaining hits'between rounds, you get detailed information about type agreement in two selected rounds.

Marky outputs

Export round data

You can obtain a printable summary of each round in the option "export results"

Export annotated documents

You can download annotated documents using the download option in the view of a Project

Generate consensus

After one round finishes you can generate the annotation consensus, either automatically or manually. Typically, the project should only be closed when a consensus is produced. Then, users may download the text annotations and the documents. The annotations are downloaded in tsv format and the documents in plain ASCII format.

Recurring doubts in project administration

The processing of agreement tables is taking too long, is it normal?

The time depends on the volume of data involving the calculation. Therefore it is recommended to choose only those that interest you at all times.

I imported a PubMed document and the contents look strange. what I can do?

Marky supports to suit most html documents, but there are exceptions.You can edit the document.If needed.

I uploaded documents from other places and do not look the same. Have I done something wrong?

This annotation tool does not reproduce original look and feel. If needed edit the document.

I would like to create projects for different domains, is it possible?

You can annotate any document that you create.

Annotators' manual

Annotation functionalities

The main menu

The annotator's menu shows the rounds open for annotation, and those which are closed and may be can consulted.

Example of the main interface of a annotator.

Clicking on the round or on the name of the project you can see its description and its associated types. If the round is not closed, you can participate in the round clicking the "Start" button.

How do I change my profile?

To make changes your user profile, you should go to menu area and click in "My Data" option.

How to annotate a document?

To begin annotating a document, you should click on the "List my rounds" and then press the "Start",

The toolbar for annotated documents

Example of toolbar for annotate a document.

In the top left corner of the page there is a series of buttons for each type or class to annotation that Marky offers. In the bottom left corner of the page's the typical paging bar for navigation between documents that can be annotated In the corner of the page is the selector that allows fast access to the document you want.

This button allows the printing of the annotated document.
This button deploys the annotation help.
This button allows you to save your work whenever you want.
This button allows you scroll to top of page.
This button allows you to search for a word in the whole text and annotate all occurrences under the same type and with the same answers.
The timer indicates the remaining time remaining to make an auto save:

Create an annotation

To create an annotation, you should click on the type of annotation corresponding to the text to be annotated. Then, select the piece of text you want, fill in the notes and press "Save".

IMPORTANT: you cannot create an annotation over text that has been annotated before.

It is recommended to create annotations in the natural sequence of reading, although it is not strictly necessary.

Edit an annotation

You cannot edit the annotation. If you wish to do this, you should delete the annotation and re-annotated this text.

To edit an annotation's notes you should place the cursor over the annotation and then press the left mouse button. Remember to press "Save" to effect this change.

Delete an annotation

To delete an annotation, you should place the cursor over the annotation, and press the right mouse button. Then a confirmation window appears and confirm your deletion.

How to see my statistics for a project?

To view statistics for a project, you should click in the menu option "List my projects." Then you choose the project view and you locate the option "My stats in this project." If you press it you annotation experience in the project.

10. Recurring doubts in annotation

Can I annotate the same text with different types?

You cannot annotated the same piece of text with two or more types.

Which browsers can I use to execute Marky annotation framework?

Marky is optimized for:

Chrome
Firefox
Internet Explorer 9+
Opera
Safari

I'm trying to edit an entry and I get an alert window. Have I done something wrong?

In this case it is recommended to delete annotation and create it again.

I deleted an entry but it is still marked, what happened?

Try reloading the document without saving.

I'm unable to print the annotated document, what I can do?

For these cases it is recommended internet search with the words "print highlighted HTML" and the name of the browser(version) you are using.

When I try to navigate the documents I get an error saying:"The users round could not be saved". What I can do?

Apparently your administrator has created an empty document. In this case it is recommended to contact the administrator to check out the problem.