From results in that study McMaster CE&B GRADE has made a Guideline Checklist site you also might want to check out. It has an online version of the common checklist and a downloadable pdf.

Essential Content development

1. Guideline panels or admins decide on the tentative Table of content structure
   This is done in the application. You add sections, make sub-sections or sub sub sections.
   See: How do I build the table of Contents
   
   
2. Guideline panels decide on the clinical questions to address and the related patient important outcomes (PICO questions)
   This is done in the application (see add a PICO and add a Outcome). The PICOs are written directly into the application in structured fields/boxes. 
   The discussion around PICO formulation can be done through the discussion features/chat room in the application. Discussion through the application can be complimented with face to face meetings/ teleconferences. Using the application for such meetings enable you to access prior written discussions and agree on items that can be recorded realtime in the app. That means everyone will have access to the updated content from the discussion even before the meeting is over. It can also be used for preparation for a meeting, by panel lead adding content the other panel members can review in advance.
   Many guideline panels will have a wish to attach codes from ontologies so that the recommendations could be linked and searched for with greater possibilities.  This is done in the application by attaching codes on a PICO by PICO basis. As an example, the ICD-10 coding could be used by software like EMRs for links to patients condition list and MeSH could be used to search for new evidence.  
   
   
3. All authors report conflict of interest on a PICO question by PICO question basis.
   Authors with primary conflicts in a specific PICO question connected to a recommendation will usually not be allowed to vote for that PICOs Evidence profile Quality assessment. They will usually also not  be able to vote on the overall quality, direction and strength of the recommendations that are based on that PICO. 
   The conflict of interest reporting will be done in the application, and plan for this feature to be present feb/march 2014.
   All authors affiliated with a PICO should state their conflict of interest as soon as the PICO is made, as this could affect the study selection and/or decision to perform new meta analysis.
   
   
4. Perform the primary literature search based on the PICO questions. The primary literature search needs to be done thoroughly, and if possibly in collaboration with librarians. 
   You can ease this process by using the information from the PICO questions that you have put into MAGICapp (both free text and MeSH, SNOMED-CT and ICD-9/10).
   The primary literature search should be done outside of the MAGIC application to allow full functionality of the various search functions of Epistemonikos, Pubmed, McPlus, Embase, Psycinfo, Pedro. For updating and alerting of new evidence however, we are building in functionality to get suggestions for new studies and updates from other knowledge repositories. Research around dynamic updating is done in the DYNAM-IT project. See Publication and updating
   
   
5. Put the references you use to the reference list, and add them sequentially to the various Evidence profiles and Recommendations. 
   The References you use needs to be placed in the main reference list. You can add details like the DOI and PMID, Abstract, PICO of study, Comments and main outcomes. Basic information like authors, journal, year and type of study will be extracted from other services like Epistemonikos or PubMed (feb 2014). We also plan for you to be be able to upload a pdf of the study and state the results in a structured PICO format. 
   When a Study is registered in the main reference list it can be added to any PICO or Recommendation. We plan for you to link to a reference to text in a free text section too, but right now you can add a reference directly to your risk estimates in the evidence profile, to a PICO as a whole or to a specific recommendation. This way the reference will only be registered once, but can be added to multiple items and places. 
   Adding the DOI and PMID to a reference makes it possible for us to create a link to the pubMed-site of the study, the Epistemonikos-site of the study and the Journals own site of the study. It also makes it possible to generate Epistemonikos missed/new studies matrix of an outcome in the Evidence profile. Having the DOI and PMID in place makes it possible to trace the reference back to the study they belong to (as there might be more than one paper coming out of a study). 
   See add content to a reference
   
   
6. Guideline panels/knowledge centers screen and select the studies to include as evidence.
   The primary litterature search will in most cases identify a lot of studies not applicable for inclusion in the evidence review, studies that do not match the panel's PICO questions. The studies must be screened and selected for inclusion. Some guideline panels use external services like knowledge centers/ Cochrane centers to do the searching, screening and selecting, and some do parts or the whole job themselves.
   We will link to the screening tools EROS and Covidence (which will consolidate through 2014) where you can do the screening of your studies directly online. This way you will have a transparent list of included and excluded studies, that would help the guideline panels in their future updates, as they would clearly see which studies they have excluded and why. In the cases where there are no updated systematic reviews/meta analysis, and more than one RCT identified from the study selection that apply to the PICO in question, the guideline panel must decide if they should make a meta analysis themselves. You can use the same screening tools to do the screening of your studies directly online before transferred to meta-analysis tools like RevMan or EPPI-reviewer 4.
   We have also initiated talks to integrate with Dr.Evidence, a web based service that provides services like making meta-analysis and doing data extraction. 
   
   
7. Making a descriptive table of the included individual studies
   In cases where there are no updated systematic reviews/meta analysis, and more than one RCT identified from the study selection that apply to the PICO in question, the guideline panel must decide if they should make a meta analysis themselves. We plan for the reference list of included references to provide the basis of an interactive descriptive table, where Outcome -information and comments regarding the individual studies/references can be visulized. The descriptive table can also include important information about the Population, Intervention, Comparator, the Outcomes and study design/follow up and number of participants. (In the future we expect a lot of this information to be automatically extracted from the references/studies, but as they are not stored with structured and tagged content in evidence databases yet, this will unfortunately be a manual job for a while longer). 
   The discussion around the description of studies can be done through the discussion features/chat room belonging to each study in the application. 
   
   
8. Making systematic reviews when needed
   The descriptive table under item 7. will provide basis for the systematic review/meta analysis process, having all the primary data and comments stored in the application for maximal transparency. 
   We will link to the screening tools EROS and Covidence where you can do the screening of your studies directly online before transferred to meta-analysis tools like RevMan or EPPI-reviewer 4. 
   Your meta-analysis (even if unpublished) should then be put in the reference list, with all the belonging data, so that you can reference it when you state your effect esimate calculations. This will ensure optimized transparency.
   The discussion around the description of studies can be done through the discussion features/chat room belonging to each study in the application. 
   
   
9. Guideline panels process the evidence and make evidence profiles
   The content of the evidence profile is written directly into the application under it's respective PICO question, in the Evidence profile-tab. You add your chosen patient important outcomes either continuous or dichotomous outcomes, and the effect estimates from the Reference/study/systematic review you have identified. Then you perform the quality assessment using the GRADE methodology, for each outcome. 
   The discussion around quality assessment can be done through the discussion features/chat room in the application. 
   Discussion through the discussion features/chat room can be complimented with face to face meetings/ teleconferences. Using the application in such meetings enable anyone in the team to access the Evidence Profile both prior to and during the meetings. When accessing the Evidence profile during the meeting, everybody would be looking at the same evidence and you could agree on items that can be recorded realtime in the app. That means everyone will have access to the updated content from the discussion even before the meeting is over.
   It can also be used for preparation for a meeting, by panel lead adding content the other panel members can review in advance.
   For difficult decisions and bigger panels, there might come a time where you would want to vote. We plan to integrate with free online voting systems like Easypolls, Addpoll and Simply Voting.
   
   You can also add practical consequences to a PICO question. This should be non-numerical issues that the patient or phycisian might care about, and that can have huge implications of patients life. It can be used as a PICO frequently asked questions, or as a 2x2 explanation of practical consequences in the style also used by our good friends behind Option Grids.
   
   Planned feature is import of numeric information from the references you have chosen, from sources like Epistemonikos, Covidence, Cochrane library, Dr.Evidence and Clinicaltrials.gov. This will reduce the workload and ensure correct data input.
   
   
10. Guideline panels write evidence summaries for PICO questions when deemed necessary.
    The evidence narrative summary is written directly into the application in the summary tab present in all PICOs. This could include summary or discussions about the evidence, the PICO formulation or the included references.
    Discussions about the content can be done through the internal discussion system/chat room.
    Discussion through the application can be complimented with face to face meetings/ teleconferences. Using the application for such meetings enable you to access prior written discussions and agree on items that can be recorded realtime in the app. That means everyone will have access to the updated content from the discussion even before the meeting is over. It can also be used for preparation for a meeting, by panel lead adding content the other panel members can review in advance.
    
    
11. Guideline panels summarize the key factors "benefit/harms, quality of evidence, preferences and values, and resource use in the Key information section.
    Under every recommendation there is a Key Information tab. Here the guideline panel should write short summaries in each of the 4 key information sections. (benefit/harms, quality of evidence, preferences and values, and resource use).
    Discussions about the content can be done through the internal messaging system/chat room.
    Discussion through the discussion features/chat room can be complimented with face to face meetings/ teleconferences. Using the application in such meetings enable anyone in the team to access the Key information both prior to and during the meetings. When accessing the Evidence profile during the meeting, everybody would be looking at the same thing and you could agree on items that can be recorded realtime in the app. That means everyone will have access to the updated content from the discussion even before the meeting is over.You can also use the application in the preparation for a meeting, eg. as a panel lead you could add suggestions for content that the other panel members can review in advance. 
    For difficult decisions and bigger panels, there might come a time where you would want to vote. We plan to integrate with free online voting systems like Easypolls, Addpoll and Simply Voting. 
    
    For each of the factors the guideline panel have the option of stating how confident they are in their statements.This is a feature to help guideline panels decide on the content and the strength of recommendation. Being a dichotomous question we believe it will spur the discussion and help in the final decision.
    Stating the confidence for each factor will also act as a visual help for the workflow, as you can clearly see where you have made a decision (green or yellow) or not yet (gray).
    This color coding will be visible for the end user, and serve as a visual cue of what made the recommendation weak or Strong. 
    
    
12. Guideline panels discuss the evidence profiles, evidence summaries and key information and come up with a recommendation and its strength. 
    The recommendations (with optional headers/remarks) are written directly into the application. The strength is also set and is reflected by a color change of the top bar of the recommendation. 
    Discussions around the recommendation can be done through the internal discussion feature/chat room. 
    Discussion through the discussion features/chat room can be complimented with face to face meetings/ teleconferences. Using the application in such meetings enable anyone in the team to access the Key information both prior to and during the meetings. When accessing the Evidence profile during the meeting, everybody would be looking at the same thing and you could agree on items that can be recorded realtime in the app. That means everyone will have access to the updated content from the discussion even before the meeting is over.
    You can also use the application in the preparation for a meeting, eg. as a panel lead you could add suggestions for content that the other panel members can review in advance. 
    For difficult decisions and bigger panels, there might come a time where you would want to vote. We plan to integrate with free online voting systems like Easypolls, Addpoll and Simply Voting. 
    
    
13. The reasoning behind the Recommendation and its strength is explained in the rationale
    Under all recommendations there is a Rationale tab. Here the guideline panel writes the Rationale in directly. See How to write Rationale.
    
    

Additional Content development

1. Adding Practical information to recommendations
   For many recommendations the guideline panel will have a need to convey practical information about the use of the recommended interventions to the end user. This can be everything from information about doses and what to do when doses are missed, to pictures, graphs and videos. 
   Writing practical information attached to a recommendation is done by writing and adding the content directly in the application, under the tab Practical Information that every recommendation has.
   
   
2. Adding Calculators and Scoring schemas to recommendations
   For many recommendations the guideline panel will have a need to attach calculators and scoring schemas. This is done in the application under the tab Practical information that every recommendation has.
   We have discussed the possibilities of linking up to services from medical calculator - applications,(eg. MedCalc) especially when the guideline is viewed on a smartphone/tablet device where being routed to a mere website is less practical
   
   
3. Adding Background information (under each section)
   You can write appropriate background text like introduction, etiologies, common issues or.. whatever you want people to know about that is not related to a specific recommendation or information belonging to a reference or an Evidence profile.
   All sections has the possibility of having added background text. Just click the background text button to open the edit panel. See: add background text to a section
   The end user will only see this background text butten where there is background text present.
   
   
4. Adding official descriptions of medications/ Information from leaflets 
   For many recommendations the guideline panel will have a wish to attach the official descriptions from National drug agencies, to decrease reproduction of information. This will be possible to do through the application, and is based on the ATC, RxNorm or NDF codes placed under the code tab (described in item 5) for each PICO. We plan to automatically link out to these services to decrease the workload for authors, but also to ensure that users always get the latest information.
   
   
5. Adding codes from ontologies to increase interoperability with other softwares 
   Many guideline panels will have a wish to attach codes from ontologies so that the recommendations could be linked and searched for with greater possibilities.  This is done in the application by attaching codes on a PICO by PICO basis. As an example, this could be used by software like EMRs for links to patients condition list and to search for new evidence.  
   
   
6. Adaptation information, if you have adapted a recommendation from another guideline
   If you have adapted a recommendation from another guideline then you should include information about what you changed, where you took the recommendation from and appropriate disclaimers.
   A more structured approach is being developed as a part of the SNAP-IT project.