{"id":4938,"date":"2014-01-14T22:50:53","date_gmt":"2014-01-15T03:50:53","guid":{"rendered":"http:\/\/www.carnaghan.com\/?p=4938"},"modified":"2018-02-11T19:34:34","modified_gmt":"2018-02-12T00:34:34","slug":"backwards-design","status":"publish","type":"post","link":"https:\/\/www.carnaghan.com\/backwards-design\/","title":{"rendered":"Backwards Design"},"content":{"rendered":"
Backwards design is a methodology used to develop curriculum, which promotes the identification of learning results at the start of the design process.\u00a0 Wiggins and McTighe defined this term as part of their Understanding by Design (UbD) framework.\u00a0 Backwards design is not a new concept, it has been around for a long time and this paper explores some of the underlying research behind Wiggins and McTighe\u2019s work.\u00a0 Backwards design promotes a focus on teaching for understanding and learning, as well as designing the curriculum in way that meets learning goals while aligning results with state requirements.\u00a0 (McTighe & Wiggins, 2012).\u00a0 Often within education, learning goals are set first, before considering the assessment strategies that will be used to measure students\u2019 understanding.\u00a0 Without considering the results that are expected of a student, the chances of goals and assessment aligning with required results, or outputs are greatly diminished.<\/p>\n
The backwards design approach is compatible with other educational models including Expeditionary Learning (EL), which is detailed in a separate qualifying exam paper.\u00a0 This paper explores the concept of backwards design within the larger UbD framework and examines real life models where its application has yielded successful results.\u00a0 The relationship between Expeditionary Learning and the backwards design approach is also discussed along with the importance of curriculum mapping.\u00a0 Some of the concepts that have been covered in more depth in the curriculum mapping qualifying exam question will be referenced in order to solidify their relationship with Understanding by Design.<\/p>\n
Understanding by Design (UbD) is a planning Framework developed by Grant Wiggins and Jay McTighe.\u00a0 The idea behind UbD is that understanding should not happen by accident, it should be designed or planned into the curriculum.\u00a0 The framework itself advocates for keeping long-term goals (student understanding) within view, while using the right blend of content, instruction and interaction to engage learners.\u00a0 Long term educational goals should be a part of the short term day-to-day lesson planning for the classroom.\u00a0 According to Grant Wiggins, there are certain design tools that can help teachers more likely reach their educational goals, regardless of their student population and other constraints.\u00a0 These tools flow from a basic design strategy called backwards design.\u00a0 Wiggins makes it very clear that backwards design refers to a backwards design approach to goals, not content.\u00a0 The end result should not be to ensure that their students covered all assigned content within the classroom, but to instead provide students with content and learning experiences they can use and apply.<\/p>\n
Backwards design is an educational methodology that promotes results<\/em>-focused design in place of content<\/em>-focused design.\u00a0 Content-focused design can be described as a process where a particular topic may be chosen to base a lesson on.\u00a0 Wiggins and McTigue (2005) used an example to describe this concept.\u00a0 Imagine a teacher bases a lesson on racial prejudice.\u00a0 A resource is then chosen, \u2018To Kill a Mockingbird\u2019, and then specific instructional methods are based on this resource and topic.\u00a0 These methods could include for example a Socratic seminar to discuss the book with cooperative groups to analyze stereotypical images in films and on television.\u00a0 The overall hope is to thereby cause learning and at the same time meet certain common core or state requirements for English language \/ arts.\u00a0 Assessment is then baked into this process in the form of a few essay questions for assessing the student\u2019s understanding of the book.<\/p>\n This approach is common within the school system and some may question what could be wrong with planning this way.\u00a0 In order to better understand the root problem, we need to take a step back and ask ourselves why we are asking the student to read this particular novel to begin with.\u00a0 What is it we are expecting the student to understand and to be able to apply in relation to the subject of racial prejudice?\u00a0 Does the student see the purpose in this resource and does it tie into the overall learning goals beyond the book?\u00a0 If the goals are met in this situation, it wouldn\u2019t have been by design, but by chance.\u00a0 The fact that the overall goal was vague to begin with in terms of specific understandings the students were to acquire, causes further problems.\u00a0 Wiggins and McTigue label this as an approach more \u201cby hope\u201d than \u201cby design\u201d.\u00a0 By simply \u201cthrowing some content at the wall and hoping it will stick\u201d will ultimately lead to failure.<\/p>\n Another set of problems are common throughout K-12 education.\u00a0 These include activity-oriented design and coverage, also known as the twin sins of design.\u00a0 Activity-oriented design refers to the flawed approach of designing learning experiences by crafting engaging activities, without considering the overall educational goals.\u00a0 Wiggins and McTighe refer to this type of design as \u2018hands-on without being minds-on\u2019.\u00a0 Coverage refers to the misguided approach of assigning complete textbooks, or entire sets of lecture material in an effort to cover all the factual material within a specified period of time.\u00a0 This is common in K-12 schools, but also particularly prevalent within higher education institutions.<\/p>\n Content-driven design and the twin sins should be avoided and instead results-driven design should be promoted within the classroom.\u00a0 \u00a0\u00a0Results-driven design drives the backwards design methodology, which first subscribes to identifying desired results of learning, and then determines acceptable evidence (assessment).\u00a0 Finally after both of these have been completed the learning experiences are planned out.<\/p>\n <\/a><\/p>\n Figure 1: Backwards Design, Wiggins and McTigue (2005)<\/strong><\/p>\n In the first stage of backwards design should involve to review the overall state common core requirements and to determine what it is that the student should be able to know, understand and be able to do in the end.\u00a0 During this stage of design, it should be decided what concepts are important as well as what enduring understandings are desired from the student.\u00a0 The main constraint during this stage is that teachers typically have more content than can be reasonably covered in a specific amount of time.\u00a0 Therefore clarity of prioritization and careful choices must be made.<\/p>\n Determining acceptable evidence ensures that we know if the student has achieved the desired results.\u00a0 The process of determining acceptable evidence encourages the curriculum planner and teacher to first think like an assessor before moving forward and designing specific units and lessons.\u00a0 This ensures that careful thought is given at the beginning on how it will be determined that students have attained the desired knowledge.\u00a0 Wiggins & McTighe (2005).\u00a0 During this phase of backwards design the following questions must be considered.\u00a0 How will we know students have achieved the desired results?\u00a0 What will we accept as evidence of student understanding and proficiency?\u00a0 In the end, students with full understanding should be able to demonstrate what Wiggins and McTighe refer to as the six facets of understanding:<\/p>\n Not all six facets are typically used together in every assessment, however they provide a guide for measurement when considering acceptable evidence.<\/p>\n After the desired results have been identified and acceptable evidence and assessments have been determined, the process of planning the individual learning experiences can begin.\u00a0 This is counter to many traditional methods of developing lessons and curriculum within education, however it is essential in order to ensure that results-driven learning happens within the classroom.\u00a0 At this stage, several things need to be addressed.\u00a0 First of all the teacher or curriculum designer must consider what enabling knowledge and skills the students will need in order to perform effectively and achieve the desired results.\u00a0 Secondly, the activities that will equip students with the needed knowledge and skills must be determined.\u00a0 What will need to be taught and coached?\u00a0 And finally the material and resources that are best suited to accomplish these goals need to be identified.<\/p>\n Wiggins and McTighe built the UbD framework with the help of previous research in the field of cognitive psychology.\u00a0 A readable synthesis of cognitive psychology and its impact on education was published in the book How People Learn: Brain, Mind, Experience and School (Bransford, Brown, & Cocking, 2002).\u00a0 This recent publication of the National Research Council summarized the past 30 years of research in learning and cognition.\u00a0 It offered new conceptions of the learning process and explained how skill and understanding in key subjects were most effectively acquired.\u00a0 A listing of the key findings from this publication that are directly relevant to UbD are listed below:<\/p>\n Summarized from McTighe (2003).<\/p>\n Understanding by Design also promotes deeper learning within the classroom.\u00a0 Deep vs Surface learning is an area of research, which has been discussed greatly over the years. \u00a0UbD fosters an environment that influences a deeper learning approach. \u00a0Surface learners tend to fall into the category of students who value the importance on \u2018minimum requirements\u2019. \u00a0They are interested only in passing the next test or reaching the \u2018needed\u2019 grade get through their course of study.\u00a0 They have little or no interest in creating and adding their own knowledge to theories or subjects in which they are learning. \u201cDeep learning involves the critical analysis of new ideas, linking them to already known concepts and principles, and leads to understanding and long-term retention of concepts so that they can be used for problem solving in unfamiliar contexts.\u201d (The Higher Education Academy, 2007).\u00a0 Deep learning can only be achieved if the student has an understanding of why they are studying the learning the material presented in their classroom.\u00a0 This is addressed in the backwards design approach by first identifying the desired results of the learning experiences and by making students aware of this.<\/p>\n There is also some overlap with UbD and the theories of Dr. Malcolm Knowles.\u00a0 While Knowles was more concerned in andragogy, adult learning theory, much of his research is also relevant to younger students.\u00a0 Knowles recommended several principles that should be applied to the design of learning experiences.\u00a0 In his first principle, Knowles demonstrated that students have a need to understand why specific things are being taught.\u00a0 Going back to the first phase of backwards design, this principle aligns closely with identifying desired results by determining what students know along with why they should know it.\u00a0 In another principle, Knowles asserted that students should be focused on problem-centered rather than content-oriented instructions.\u00a0 Here we can see a direct alignment with the UbD framework that identifies one of the twin sins of design, coverage based education, should be avoided.\u00a0 Orienting lessons towards content coverage is a misguided approach and should be avoided.\u00a0 Knowles also argues that adults are self-directed and want to discover things for themselves.\u00a0 This is supportive of younger learners too within both UbD as well as Expeditionary Learning education.<\/p>\n Wiggins and McTighe\u2019s first publication on Understanding by Design provides an effective model for developing the curriculum using the backwards design approach.\u00a0 It provides templates and the book format begins with a high level view of the curriculum and works through to a more detailed structure and templates.\u00a0 Chapter 12 of the book focuses on what Wiggins and McTighe describe as the \u2018big picture\u2019 of using UbD as a curriculum framework.\u00a0 Earlier chapters included detailed information on developing and crafting understanding at the unit level.\u00a0 The curriculum framework examines the school district\u2019s overall goals for learning and aligns these with content standards.\u00a0 The framework is directly compatible with the curriculum mapping work of Jacobs (2004), covered in more depth in the curriculum mapping qualifying exam paper.<\/p>\n The Understanding by Design Guide was published by Wiggins and McTighe a number of years after their initial UbD book.\u00a0 The goal of this publication was to provide a model that teachers and school leaders could use to design high quality UbD education.\u00a0 While the first book provided templates and an overview of what UbD is, Wiggins and McTighe felt it did not go far enough in providing a guide to actually planning for understanding and designing quality essential assessment questions.\u00a0 This publication also came with more in depth templates for developing a robust UbD curriculum.<\/p>\n Shumway & Berrett (2004) published the results of a modified approach to backwards design carried out at Brigham Young University.\u00a0 The purpose was to improve their students\u2019 knowledge of the new Standards for Technology Literacy (STL) and to help implement them within local schools.\u00a0\u00a0 The students in the study were in the teacher education program at the university and a strategy was devised to help educate them on the standards, while at the same time putting them into practice at schools where they participated in teacher training.\u00a0 Several problems were identified early on.\u00a0 The majority of teachers within the local schools had a copy of the standards and were supportive of them, however they needed support for implementing them within the classroom.\u00a0 Much of the help needed was to ensure that curriculum units designed for their classes were standards based and not merely standards reflective.\u00a0 At the university, teacher education students needed to have earlier and more frequent teaching and curriculum development experiences in the public school.\u00a0 Students needed to become intimately familiar with the need for these standards and learn how to develop curriculum based on them.\u00a0 The university needed to coordinate efforts and strengthen the partnership between the schools and local classroom technology teachers.\u00a0 All of this led to the development of an initiative using a modified version of backwards design to guide the process.<\/p>\n The reason a modified version of backwards design was used happened due to challenges the students ran into while trying to implement it.\u00a0 The students expressed frustration when trying to determine assessment procedures when no unit or lesson plans had been developed.\u00a0 In order to overcome this challenge, the first modification was implemented in the form of determining acceptable evidence at a more general level (e.g. portfolios, projects, etc.) in order to provide specifics later on after learning experiences had been planned.\u00a0 Another phase called enrichment was introduced, which allowed for more collaboration and formative evaluation of the work carried out between the university students and supervising teacher.\u00a0 Once the unit plan, lesson plans, assessments, and other materials had been developed, the students submitted them to the cooperating teacher for feedback.\u00a0 With the cooperating teacher\u2019s supervision, teams of students then co-taught a short lesson related to the current sixth grade unit based on the Standards for Technology Literacy (STL).\u00a0 In later years, students would return to teach an entire curriculum unit with local high school teachers.\u00a0 The last phase of the modified backwards design model involved evaluation of the curriculum that would happen along with continual refinement.\u00a0 The modified approach served both Brigham Young University and the local district well for their unique purpose that required collaboration between the two different educational populations.<\/p>\n Backwards design has become a very popular model for designing and implementing curriculum throughout many schools and districts.\u00a0\u00a0 There are however other curriculum models, which have gained popularity over the years that should also be considered when devising any type of curriculum strategy.<\/p>\n ADDIE stands for Analysis, Design, Development, Implementation and Evaluation, which describes the various steps of the model.\u00a0 The ADDIE model provides designers with the necessary structure for designing any curriculum.\u00a0 Hodell (2006).\u00a0 ADDIE has been revised and modified many times over the years and still remains a popular framework for designing instruction.\u00a0 The ADDIE model is very similar to backwards design whereby its analysis phase is similar to identifying desired results of learning.\u00a0 Much of the initial planning for the rest of the design process takes place here. \u00a0\u00a0Analysis should define measurable terms and the overall desired level of performance in order to define the overall learning experience defined in the backwards design model.<\/p>\n ADDIE ensures that a rigorous analysis is carried out, followed by design, development and implementation of lessons and instruction that yields measurable results.\u00a0 The entire process is cyclical where the evaluation phase requires revisiting and improvement on design and development activities to ensure the finished lessons or units meet the goals of the institution.\u00a0 While much of ADDIE is similar to backwards design, it lacks certain elements of UbD including self-evaluation techniques such as student reflections of rethinking and revisiting concepts.\u00a0 UbD is also more closely aligned with K-12 school environment, while ADDIE is commonly used in higher education.<\/p>\n The ASSURE model was developed by Heinrich and Molenda in 1999. It is a well-known instructional design guide using constructivist perspective which integrates multimedia and technology to enhance the learning environment.\u00a0 Faryadi (2007).\u00a0 ASSURE is another curriculum model that uses the letters in its name to represent the various processes involved.\u00a0 \u2018Analyze the learner\u2019 is the first phase where the model encourages the designer to gain an understanding of the student and their needs.\u00a0 \u2018State objectives\u2019 is the second phase, which become the overall focus of the lessons to follow.\u00a0 \u2018Select media and materials\u2019 comes next to ensure the correct resources are chosen for the type of instruction being designed, this is followed by \u2018utilize media and materials.\u2019\u00a0 \u2018Require learner performance\u2019 is the next phase, which requires the formation of strategies to encourage student participation in the unit or lesson.\u00a0 Evaluation and revision are both covered in the last phase, which refers to both evaluation of teaching and well as student achievement.<\/p>\n The ASSURE model draws similarities from the ADDIE model, however there are also a few similarities to the backwards design approach.\u00a0 One issue of the model is that evaluation is left to the end.\u00a0 While it is helpful to know what worked and what didn\u2019t, at this stage of the design process it may be too late to intervene for student success.\u00a0 Another point to note with ASSURE is its heavier reliance on technology resources, something the other models do not explicitly mandate.<\/p>\n There are other approaches to curriculum development and instructional design that go outside the scope of this paper.\u00a0 Models such as the 5E Instructional Design Model, the Dynamic Instructional Design Model, and Gagne\u2019s Model are just some examples of other approaches to developing instruction.\u00a0 One other method or approach of designing curriculum is important to address, which can be used in combination of the models covered in this paper.\u00a0 This method is called bottom-up or bottom driven approaches to designing and implementing curriculum changes within the school.<\/p>\n Bottom up and top down can be used to describe different things.\u00a0 One meaning of bottom up for example describes the design and development of a lesson or unit by starting at the lowest level or individual parts.\u00a0 Top down on the other hand starts at the highest level or \u2018the big picture\u2019.\u00a0 Another interpretation of top down and bottom refers to the way curriculum is planned and implemented within schools and districts.\u00a0 Traditionally a top down approach in this sense has been more common where initiatives come from the top or school leaders and directed to the teachers to implement.\u00a0 In recent years, bottom up strategies, those lead by teachers, have become more common with frameworks such as UbD that encourage teacher involvement in the design of units and lessons.<\/p>\n McCarthy (2009) discusses the benefits of this approach and asserts that the bottom-up approach helps to promote changes at the grass-roots level, which in turn has led to more consideration of the needs of learners.\u00a0 At the same time decisions made about curriculum development have encouraged teacher professional growth where bottom-up strategies have been implemented.\u00a0 McCarthy focused on curriculum reform in her paper, and her study found that bottom-up methods ensured success of this reform because of buy-in from the teachers where they were made aware of their power as change agents.\u00a0 While there needs to be a balance of direction from the top along with grass roots involvement, backwards design promotes teacher collaboration, which ensures that appropriate lessons are developed to meet the overall goals of the school, district and common core requirements.<\/p>\n Understanding by Design is directly compatible with the Expeditionary Learning (EL) approach to education, covered in more depth in the curriculum mapping qualifying exam paper.\u00a0 EL and UbD complement each other by first examining standards, or results that provide a bigger picture in terms of the overall learning goals for students.\u00a0 They both promote deeper understanding through learning experiences, critical thinking, and going beyond surface based and basic learning patterns.\u00a0 Students should be able to take away a thorough understanding of the \u2018big ideas\u2019 instilled in them through the process of an effectively designed curriculum.\u00a0 These big ideas should contain the knowledge and experiences that will stay with a student far beyond high school graduation.<\/p>\n While UbD and EL complement each other, a third component or toolset can be effectively added to the mix in order to provide the right resources to effectively record, manage and maintain a standardized curriculum.\u00a0 This toolset or process is known as curriculum mapping, also covered in more depth in the curriculum mapping qualifying exam paper.\u00a0 Curriculum mapping is integrated into the Expeditionary Learning Core Practices.\u00a0 In their publication regarding underlying research in UbD, Wiggins and McTighe cite several cases where UbD influenced curriculum mapping practices at several schools.\u00a0 An existing curriculum mapping framework developed by Heidi Hayes Jacobs in the late 90s has become a gold standard of mapping within public schools.\u00a0 It is compatible with UbD and EL approaches and can be used to provide an effective platform to organize, maintain and communicate curriculum design throughout the school and district, while playing a role to ensure standards are correctly aligned to these approaches.<\/p>\n Understanding by Design and the backwards design approach provide teachers with a methodology to begin planning the curriculum alongside common core requirements.\u00a0 The framework begins by first identifying the results that students should experience at the end of the lesson.\u00a0 It differs from many traditional approaches to designing instruction where assessment is planned early on before the learning experiences or actual unit is finalized.\u00a0 This ensures teachers continue to design and develop results-driven lessons, which can be measured against a defined set of requirements.<\/p>\n Wiggins and McTighe place a strong emphasis on understanding that goes beyond the classroom.\u00a0 This approach is compatible with other instructional theories including the adult learning research carried out by Dr. Malcolm Knowles.\u00a0 It is also directly compatible with Expeditionary Learning that values experience and understanding.\u00a0 Both UbD and EL can harness the power of an effective curriculum mapping process guided by the work of Dr. Heidi Hayes Jacobs.<\/p>\n Baran, B. (2010). Experiences from the Process of Designing Lessons with Interactive Whiteboard: ASSURE as a Road Map. Contemporary Educational Technology<\/em>, 1(4), 367\u2013380.<\/p>\n Bransford, J., D., Brown, A., L., & Cocking, R., R. (2002).\u00a0 How People Learn: Brain, Mind, Experience, and School<\/em>.\u00a0 National Academies Press: Atlanta, GA.<\/p>\n Blonder, R., Kipnis, M., Mamlok-Naaman, R., & Hofstein, A. (2008). Increasing Science Teachers\u2019 Ownership through the Adaptation of the PARSEL Modules: A \u201cBottom-up\u201d Approach. Science Education International<\/em>, 19(3), 285\u2013301.<\/p>\n Chevalier, R. D. (2011). When did ADDIE become addie? Performance Improvement<\/em>, 50(6), 10\u201314. doi:10.1002\/pfi.20221<\/p>\n Expeditionary Learning (2011).\u00a0 Expeditionary Learning Core Practices.\u00a0 A Vision for Improving Schools<\/em>.\u00a0 City, State: Expeditionary Learning.<\/p>\n Faryadi, Q. (2007). Instructional Design Models: What a Revolution! \u00a0ERIC Online Submission.<\/p>\n The Higher Education Academy (2007).\u00a0 Deep and Surface Approaches to Learning.\u00a0 Retrieved from: http:\/\/exchange.ac.uk\/learning-and-teaching-theory-guide\/deep-and-surface-approaches-learning.html<\/p>\n Hodell, C. (2006).\u00a0 ISD From the Ground Up<\/em>.\u00a0 Alexandria, VA: ASCD<\/p>\n Jacobs, H., H. (2004). Getting Results with Curriculum Mapping<\/em>.\u00a0 Alexandria, VA: ASCD.<\/p>\n Jacobs, H., H. (1997). \u00a0Mapping the Big Picture. Integrating Curriculum & Assessment K-12<\/em>.\u00a0 Alexandria, VA: ASCD.<\/p>\n Knowles, M. S., Holton III, E. F., Swanson, R., A. (2005). The Adult Learner<\/em>. 6th Edition. London, UK: Elsevier Butterworth Heinemann.<\/p>\n McCarthy, M.\u00a0 (2009). Implementing curriculum change: A \u201cbottom-up\u201d approach. (n.d.). Retrieved June 12, 2013, from http:\/\/www.academia.edu\/2065020\/Implementing_curriculum_change_A_bottom-up_approach<\/p>\n McTighe, J., & Seif, E. (2003).\u00a0 A Summary of the Underlying Theory and Research Base for Understanding by Design.\u00a0 Retrieved from: http:\/\/assets.pearsonschool.com\/asset_mgr\/current\/201032\/ubd_myworld_research.pdf<\/p>\n Newmann, F. M., Marks, H. M., & Gamoran, A. (1996). Authentic Pedagogy and Student Performance. American Journal of Education<\/em>, 104(4), 280\u2013312. doi:10.2307\/1085433<\/p>\n Shumway, S., & Berrett, J. (2004). Standards-Based Curriculum Development for Pre-Service and In-Service: A \u201cPartnering\u201d Approach Using Modified Backwards Design. Technology Teacher, 64(3), 26.<\/p>\n Wiggins, G., & McTighe, J. (2005). Understanding by Design, Expanded 2nd Edition. Association for Supervision and Curriculum Development.<\/p>\n Wiggins, G., & McTighe, J. (2012).\u00a0 Understanding by Design Framework.\u00a0 ASCD Whitepaper.\u00a0 Retrieved from: http:\/\/www.ascd.org\/ASCD\/pdf\/siteASCD\/publications\/UbD_WhitePaper0312.pdf<\/p>\n","protected":false},"excerpt":{"rendered":" Backwards design is a methodology used to develop curriculum, which promotes the identification of learning results at the start of the design process.\u00a0 Wiggins and McTighe defined this term as part of their Understanding by Design (UbD) framework.\u00a0 Backwards design is not a new concept, it has been around for a long time and this […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7128],"tags":[4212],"post_series":[],"class_list":["post-4938","post","type-post","status-publish","format-standard","hentry","category-ux-design","tag-academic-papers"],"yoast_head":"\n<\/a>1. Identify Desired Results<\/h4>\n
<\/a>2. Determine Acceptable Evidence<\/h4>\n
\n
<\/a>3. Plan Learning Experiences<\/h4>\n
<\/a>Supporting Research<\/h3>\n
\n
<\/a>An Effective Model for Mapping the Curriculum<\/h3>\n
<\/a>Curriculum Design with UbD in Action<\/h3>\n
\n<\/a>Figure 2: Modified Backwards Design, Shumway & Berrett (2004)<\/strong><\/p>\n<\/a>Other Curriculum Models<\/h3>\n
<\/a>The ADDIE Model<\/h4>\n
<\/a>The ASSURE Model<\/h4>\n
<\/a>Bottom up Approach to Curriculum Design<\/h4>\n
<\/a>UbD, EL and Curriculum Mapping<\/h2>\n
<\/a>Conclusion<\/h2>\n
<\/a>References<\/h2>\n