NURS 6431: Evaluation Methods for Health Information Technology | Week 7

Imagine you have recently started a new job as the first nurse informaticist to be employed in a large community clinic. To prepare you for your duties, you receive the following job description:

Your primary responsibility will be the implementation of health informatics technology systems and all necessary support processes. Your supplemental responsibilities will include developing systems to meet stakeholder desires, planning personnel training, and maintaining current systems. In addition, you will be responsible for implementing and maintaining quality initiatives. All of these duties should be completed in a timely fashion and within budget. NURS 6431: Week 7: Evaluation Methodology Planning Paper

 

On the first day of work, you are shown to your cubicle, and no further instructions are given. When you attempt to obtain further information, management states that you are in a new position and that they are still unclear about your role. How would you feel? Where would you start? Think of the details not included—such relevant information as what systems are currently in use, the identity of key stakeholders, budgets, and current plans—and how their absence influences your ability to be effective as a nurse informaticist.

The situation above indicates a lack of planning on the health care organization’s part. The organization’s leadership decided hiring a nurse informaticist would be useful but lacked a clearly defined methodology for integrating the field of informatics into their organization. This lack of methodology has promoted a rushed, expensive, and poorly understood hiring and onboarding process. Had the leadership developed a clear methodology, they could have minimized waste and improved understanding.

The same clarity is essential when designing the actual methodology for implementing an evaluation plan. All details of the evaluation procedure should be carefully identified, and the evaluation methodology should be written in unambiguous language. Someone unfamiliar with the project or process should be able to gain a clear picture of what the evaluation addresses and how it will be conducted simply from reading the plan. This week, you consider evaluation methodology planning and what is involved in creating a plan that is thorough and focused.

Learning Objectives – NURS 6431: Week 7: Evaluation Methodology Planning Paper

Students will:

• Analyze the characteristics of strong and weak evaluation methodologies
• Analyze the process of developing an evaluation methodology plan from a PICO question
• Create an evaluation methodology plan for a PICO question*

* The Assignment related to this Learning Objective is introduced this week and submitted in Week 8.

Learning Resources

Required Readings

Friedman, C. P., & Wyatt, J. C. (2010). Evaluation methods in biomedical informatics (2nd ed.). New York, NY: Springer Science+Business Media, Inc.

• • Chapter 4, “The Structure of Objectivist Studies” (pp. 85–112)This chapter examines the key concepts in relation to the design of studies and the measurement of results. It includes definitions for fundamental terms, a discussion on the levels of measurement, and a description of the different types of study designs.

• Chapter 9, “Subjectivist Approaches to Evaluation” (pp. 248–266)This chapter introduces the subjectivist approach to evaluation and highlights the key ways it differs from an objectivist approach. The chapter also examines the premises upon which this type of study is based, and how qualitative data are recorded and analyzed.

Centers for Disease Control. (n.d.). Evaluation planning: What is it and how do you do it? Retrieved from http://www.cdc.gov/healthcommunication/research/evaluationplanning.pdf

This document provides a brief overview of planning an evaluation, including the different types of evaluations and the components needed in developing the evaluation methodology.

Stroud, S., & Gansauer, L. (n.d.). Nursing evidence-based nursing practice tool kit: Practice, evidence, and translation process. Spartanburg Regional Health Care System.

This paper provides guidelines for conducting an evaluation. It highlights the different phases of conducting an evaluation and the steps included in each phase.

Discussion: A Critique of Evaluation Methodology Plans

Developing a relevant PICO question that accurately addresses the goal of an evaluation and then locating the most current information on the topic are both key steps in the evaluation process; however, of equal or greater importance is the development of the methodology to gather the data that will answer the PICO question. This is where the evaluator must determine the “who,” “what,” “when,” “where,” and “how” of the evaluation. The evaluation methodology outlines the specific steps that will be taken to complete the evaluation. Who will be involved? What sort of research design should be used? Where is the evaluation taking place? How much time will the evaluation require, and how many participants are needed? How will the evaluation be conducted? It is imperative that the evaluator takes the time to make sure a methodology plan is clear, specific, and thorough.

In this Discussion, you critique a series of poorly constructed evaluation methodology plans, identify areas of weakness, and recommend how they can be improved.

The following scenarios will be used for this Discussion:

Scenario #1: Agnes, the nurse informaticist at a small rural hospital, has been asked to develop an evaluation plan to determine the success of an upcoming training program for the launch of a new computerized nursing documentation system. Agnes has developed the following methodology plan:

“I will speak to participants immediately after the training program to determine the success of the training. They will be asked about the instructor, if the training was a good time length, if there were enough breaks, and if the training location was comfortable. After the implementation, I will ask the physicians and nurses if they like using the new nursing documentation system and how much time it saves them weekly.”

Scenario #2: Maria, a nurse informaticist in a large surgical center, has been asked to develop an evaluation of the implementation of a new Operating Room Management System (ORMS) that includes scheduling, case cart management, and surgical case documentation. Maria has developed the following evaluation methodology plan:

“I will conduct a 30-minute interview with each nurse in the surgical ward to determine his or her impressions of the new ORMS. I will ask them to specify how they log into the system, to detail how often they use it each day, to describe what types of information they utilize, and to provide a detailed list of issues they encounter. I will have the nurses rank 50 different characteristics of the ORMS on a 1 to 100 scale. In addition, I will ask each surgeon to document his or her impressions of the case documentation functions.”

Scenario # 3: The CEO of the hospital system in a major metropolitan area is a brusque, hard-to-please individual. Carl, a newly hired nurse informaticist, has been tasked with developing an evaluation to correspond with the implementation of a health analytic system that the CEO has hand-picked. Carl has developed the following evaluation methodology plan:

“I will arrange one morning where groups of three nurses at a time will have a 15-minute, face-to face meeting with the CEO to both answer his questions and discuss their experiences using the new health analytic system tool. By having this candid dialogue, but without structured questions or parameters, a good overall understanding of the value of the analytic system should be obtained.”

To prepare for NURS 6431: Week 7: Evaluation Methodology Planning Paper:

• Review the three evaluation methodology plans outlined within the scenarios above.
• Critique each plan. Is it concrete? Is it specific? What are the strengths? Weaknesses?
• Based on this week’s Learning Resources, recommend at least two changes that would strengthen each plan.
• Research the Walden Library to find an example in the literature of an evaluation study that has a strong evaluation methodology plan, and assess why you believe it to be strong.
• Consider your own PICO question and the elements that would need to be included in the methodology plan to adequately answer this question.

 

By Day 3

Post a brief critique of each of the evaluation methodology plans. Describe how each could be strengthened. Briefly summarize the evaluation study you identified in the Walden Library (include the reference in proper APA format), and explain the elements that made you conclude it has a strong methodology component. Describe how you can utilize what you have observed in both the poor and the strong methodology evaluation plans to ensure that you develop an appropriate methodology to answer your PICO question. Outline specific elements that would need to be clearly identified in your evaluation methodology, and explain why they are important to include.

By Day 6

Respond to at least two of your colleagues on two different days using one or more of the following approaches:

• Share an insight from having read your colleagues’ postings, synthesizing the information to provide new perspectives.
• Validate an idea with your own experience and additional research.
• Expand on your colleagues’ postings by providing additional insights or contrasting perspectives based on readings and evidence. NURS 6431: Week 7: Evaluation Methodology Planning Paper.

 

 

ADDITIONAL INFO 

A Critique of Evaluation Methodology Plans

Introduction

Evaluation methodology plans are a critical component of your evaluation. They provide an outline of what you will do to gather the data, how it will be collected and analyzed, and who will be involved in those activities. These plans should establish a clear workflow that leads from concept through execution to final publication of findings.

Research Objectives

It is important to state the purpose of your study before you begin. State what you want to know, and why. For example, if you are conducting a research project on the relationship between technology adoption and job performance (i.e., how well employees use computers in their work), then it might be helpful to clarify that goal as: “To determine whether there is an association between computer use at work and job performance among office workers”

The second step after establishing goals is planning your research design; this will help keep everything organized so that no questions arise later on in the process that might result from having not planned ahead properly.

Research Questions

The research questions are a way to focus your evaluation. They help you decide what data to collect and help ensure that you’re answering the right questions. The research questions should be clear, concise and relevant.

Research question: What do parents think about their child’s school?

In order for this study to be valid, we need an answer for our first question (the “problem”) as well as one for our second (the “desired outcome”). For example: A parent may say that they want their child’s teacher to know how well their child is learning because they feel like she’s not teaching them properly or efficiently enough; or another parent might say that he doesn’t like when teachers don’t communicate with parents at all times during class time, which causes him stress due having nothing going on behind closed doors at end of lesson period etc… Both statements could mean different things depending on context but both still fit under one umbrella category such as: inadequate communication between teacher/students regarding needs/needs not being met by either party involved in any way shape or form!

Data Collection Methods

• Inquiries

• Interviews

• Observations

• Surveys

• Focus groups and other types of group work can be an effective way to gather information, but they require careful planning and management. For example: the people who participate in the study must be carefully selected so that there is sufficient diversity; the questions should be worded so as not to bias or stereotype any participant; a moderator should be present at all times (to ensure that everyone has an opportunity for participation); participants must not feel pressured into saying something they don’t want or need; notes taken by observers should be kept confidential until they are shared with interviewers; participants should know their rights before agreeing on anything related to confidentiality agreements made during interviews/focus groups etc

Data Measures

Data measures are the variables you will use to collect information. They can be qualitative or quantitative, and they include things like observation, interviews, surveys, and focus groups. Qualitative data measures include things like observation; however you need a source of numerical information for your evaluation plan so that you can quantify it (for example: test scores). Quantitative data measures include tests and surveys.

Population to Be Studied

The first step in the evaluation methodology planning process is to clearly define the population to be studied. This can be done by answering three questions:

• What type of people will you study?

• How many people are you going to study?

• Where are they located (a geographical location)?

Sampling Procedures and Rationales

The sampling procedure should be based on the purpose of your study and its reliability. For example, if you want to know how many people are likely to purchase a product or service, then a random sample would be appropriate. However, if you need data on what products are selling well in your area and why they’re selling so well (or not), then non-random stratified sampling may be more appropriate.

A sampling method also has implications for validity: if there are too few participants in one group (for example those who have been referred by an existing customer base), then this could affect the results of an analysis made after excluding these participants from the sample pool because they may have skewed the sample toward certain types of people who looked at this website more often than others did.

Limitations of the Study

The study has some limitations. The most important limitation is that the sample is limited to only one university. Another limitation is that it was not possible to conduct an independent evaluation of all four institutions because they are located within a single city and share resources and facilities, such as libraries and computer labs.

In addition, the study does not include all aspects of evaluation methodology planning in its scope; for example, there are other steps involved in conducting evaluations such as identifying stakeholders or determining how best to measure success (e.g., through surveys).

Justification of Resources and Budget Needs

One of the first things a project manager must do is justify the use of resources. In other words, what are your budget needs? How much money will you need and how will it be spent?

A good way to do this is by comparing your project’s costs with its benefits. For example, if you’re designing an app for a homeless shelter that helps people find shelter in their area and connect them with services, then one benefit might be helping them get off the streets or out of shelters. Another benefit might be helping them get jobs so they can pay rent on whatever place they end up living at (which could mean paying for one night). A third benefit could be reducing crime rates in neighborhoods where there are many homeless individuals who don’t have access to safe places or help seeking out employment opportunities during daytime hours when businesses open up shop again after being closed due to bad weather conditions caused by hurricane Irma last year.”

Takeaway:

If you’re like me, you may have been reading this article with the expectation that it would provide practical advice for improving evaluation methodology plans. However, if you’ve been following along with our journey so far and have experienced some difficulties or challenges along the way, then perhaps this isn’t what you were expecting! Let’s take a step back and look at some of the broader implications of what we’ve discussed so far:

• Evaluating qualitative research can be difficult because there is no clear way to measure how effective a particular technique was in terms of producing valid results (e.g., whether participants felt satisfied).

• Evaluating quantitative methods requires more complicated statistical calculations than those used in qualitative research because these types of studies tend not only deal with human behavior but also include variables such as time scales involved within them.

Conclusion

We hope this critique of evaluation methodology plans has been helpful in your efforts to develop a successful plan. We encourage you to keep the main points in mind as you prepare for your next round of research.