Syntax Literate: Jurnal Ilmiah Indonesia p�ISSN:
2541-0849 e-ISSN: 2548-1398
Vol. 7, No. 12, December
2022
A WELL-UNDERSTOOD SURGICAL INFORMED CONSENT
: A SCOPING REVIEW
Rieke Cahya1, Diah
Melly Maryana1, Adik
Wibowo2
1Hospital
Administration Studies Program, Faculty of Public Health Universitas Indonesia,
2Health Policy and
Administration Department Faculty of
Public Health, Universitas Indonesia
E-mail: [email protected]
The implementation of surgical consent has shifted from simply getting a signature to a focus on doctor-patient communication. Providing adequate information is very important for patients in making decisions so that patients do not feel forced to agree. Good understanding is required before a patient gives consent, but many patients have difficulty understanding and doctors fail to provide correct information. The aim of this scoping review is to investigate and evaluate patient comprehension of surgical informed consent provided by standard procedure or other methods and factors affecting patient comprehension. Three electronic database searches (ProQuest, ScienceDirect, and Scopus) were performed from 2017 until 2022. A total of 391 articles were identified and 25 articles were selected according to the PRISMA guidelines and the PCC framework. 9 of 11 articles stated that patient understanding level was low. Factors that influence it include education level, age and language limitations. 12 studies (85%) showed that patient understanding improved with the use of additional information media. Overall patient comprehension of surgical informed consent is low. This depending on the communication between the doctor and the patient. An interesting interventions can be used to improve patient understanding of surgical informed consent.
Keywords:
Patient Comprehension, Patient
Knowledge, Patient Understanding, Surgical Informed Consent.
Introduction
Patient
safety in healthcare has received significant attention over the last few
decades. The attention is a form of awareness that humans must have made
mistakes. However, in the development of health services, a system has been
created that aims to reduce and prevent errors. The implementation of this
system is based on the understanding that the health services provided must
focus on patient, service and patient safety. One application in the health
care system is well-understood informed consent.(Khan, 2012; Mudiyanselage, Eshana, Mathangasinghe,
Sarath, & Banagala, 2019; Richardson, 2013) 'Informed consent'
is the process of providing information to the patient as a consideration for
the patients to make a voluntary decision to undergo or refuse medical
procedure.(Pallett et al., 2018; Villanueva, Talwar, &
Doyle, 2018)
The recognition of informed consent has expanded significantly not only with regard to medical practice but also to clinical research. This began from a legal decision based on the principle of patient autonomy in 1905. However, the principle of "informed consent" remained unknown until in 1957 a legal decision that identified and focuses on the need to inform patients about the benefits and risks of each documented medical procedure.(Bazzano, Durant, & Brantley, 2021; Cocanour, 2017) It is stated that the patient must have sufficient knowledge and understanding as a reference in making the right decision. This statement identified several important items, namely the information provided and the patient understanding. Information is defined as an explicit explanation to the patient about the diagnostic or therapeutic medical procedure that will be given and contains specific elements that contain the purpose of the procedure, benefits, risks, steps to overcome risks, alternatives and opportunities for the patient to ask questions and allow the patient to withdraw at any time from the procedure to be performed.(Bazzano et al., 2021) The purpose of presenting �key information� is to facilitate comprehension. Comprehension become one of the important issues in informed consent before the patient gives consent.(Bazzano et al., 2021)
Currently, informed consent has become an ethical concept as outlined in the law on medical practice.(Cocanour, 2017; Khoshrang et al., 2021; Moeini, Shahriari, & Shamali, 2020) It is stated that informed consent is more than just signing legal document. Informed consent facilitates the patients to be actively involved in the decision-making process on medical procedure.(Lin et al., 2017) We need to remember that not all patients who agree and sign the informed consent document have understood all the information explained by the doctor.(Dewar, Pieters, & Fried, 2021) The three basic criteria needed to obtain consent for a planned medical procedure are that the patient must be competent, adequately informed and not coerced.(Cocanour, 2017; Lin et al., 2019)(Pucher et al., 2019) They must be able to understand the information provided, inform their decisions and understand the consequences of the decisions they make. The Physician's efforts to provide adequate information and to assess the patient's understanding are a critical measure for the success of the informed consent process.(Cocanour, 2017; Lin et al., 2017)
������������������ Providing information about medical procedures in medical terms that are difficult for patients to understand can cause anxiety and confusion for patients, especially surgery which could be a frightening experience. Limited time in communication due to urgency of the operating process, acute and vulnerable condition, emotional stress, physical pain due to injury and fear of planning surgery affect the patient's ability to listen and interfere patient comprehension of the information provided by the physician.(Agozzino et al., 2019; Lin et al., 2017, 2019; Nuraeni, 2016)
It is a challenge for health workers, especially surgeons, to provide important information about surgical plans in lay language that is easy for patients to understand. Most patients find it difficult to understand medical terms and clinical concepts used by surgeons, this is often not realized and is the biggest cause of a surgeon's failure to provide information to patients.(Villanueva et al., 2018) These conditions are challenging especially understanding that a good informed consent process can improve the patient-physician relationship.(Khoshrang et al., 2021) Accordance to various methods of delivering informed consent, the level of patient understanding that is formed also varies. The language barrier, the patient's education level and the doctor's ability to communicate affect the patient's understanding of informed consent. These three things are the most common barriers to communication between doctors and patients.(Cocanour, 2017; Fink et al., 2010; Fudman et al., 2019)
However, the process of implementing informed consent and the patient understanding of informed consent has not fully achieved the expected results. An American study conducted by Hall et al concluded that sixty-nine percent of patients decided to undergo surgery even before seeing surgeon, and 47% patients stated that the surgeon did not influence their decision. Most (68%) patients consider informed consent only as a legal formality and have little influence on the decision-making process.(Hall et al., 2012) Many medical consent forms fail to be a simple and clear communication because the language contained in them is difficult for patients to understand.(Zimmermann, Pilarska, Gaworska-Krzemińska, Jankau, & Cohen, 2021)
In Indonesia, not much research has been done on surgical informed consent, especially regarding the patient understanding. Ulfa et al in their research revealed that the implementation of surgical informed consent had not been properly carried out, the completeness and accuracy in filling out informed consent documents had not reached 100%, the contributing factor was the low awareness of medical personnel in charge of carrying out informed consent properly.(Ulfa, 2018) Herfiyanti et al in their research at Hasan Sadikin Hospital Bandung stated that the medical information provided by doctors was incomplete, which was in the explanation of prognosis (54.1%), alternatives & risks (52.5%), and complications (50.8%).(Herfiyanti, 2015) Susanto et al in their research in the city of Malang stated that the patient's ability to understand informed consent was very low and necessary to create a media by the hospital to facilitate health workers in providing medical information to patients and to help improve patient understanding.(Susanto, Pratama, & Hariyanto, 2017) Good results were obtained by Alvionita et al, specifically the explanation of informed consent in elective surgery patients at Pirngadi Hospital Medan went smoothly, detailed explanations had been given by the doctor in accordance with the provisions contained in the regulations and filling out the informed consent in accordance with the procedure.(Alvionita, Harahap, & Aini, 2021)
Failure
to implement informed consent gives the conclusion how difficult to
get the patient understanding of the information regarding the surgery given.
Many questions arise from this statement, about how is patient comprehension
regarding surgical informed consent, factors affect that patient comprehension
level and what are the challenges in the informed consent process, and whether
the use of additional information media can improve patient comprehension of
surgical informed consent. By answering the question, a doctor
can determine the best strategies and methods that can be used to provide
information about the planned surgery. The aims of this scoping review are to investigate and evaluate patient comprehension of
surgical informed consent provided by standard procedure or other methods and to
determine factors affecting patient comprehension.
The selection of this scoping review was based on the consideration that
a scoping review is a suitable method for summarizing findings that is
heterogeneous in a research method. A literature search of the ProQuest,
Science Direct and Scopus databases was conducted in 2017 until 2022. The
search strategy for each included database is presented in table 1. The
following search terms included all identified keywords :
patient
comprehension, patient understanding, patient knowledge, surgical
informed consent; and Bolean operators �OR� and
�AND� to ensure a far-reaching capture
of existing works. This framework for scoping reviews was compiled with the
Joanna Briggs Institute Methodology for scoping reviews.
The following question based upon the PCC (Population, Concept and
Context) are :
Population ����� :
Adult surgical patient
Concept ��������� : Patient's understanding of the informed consent given by
the doctor
Context ��������� :
Hospital
Tabel 1.
Search
Strategy
Database |
Keywords
And Boolean |
Proquest |
((MAINSUBJECT.EXACT("Knowledge")
OR MAINSUBJECT.EXACT("Acceptance") OR comprehension) AND
(MAINSUBJECT.EXACT("Informed consent") AND
MAINSUBJECT.EXACT("Surgery"))) AND (at.exact("Article")
AND stype.exact("Scholarly Journals") AND
pd(20170521- 20220521)) |
Scopus |
Comprehension Or Understanding
Or Knowledge And
Patient
And Informed And
Consent
And Surgical* And
Hospital
And ( Limit-To ( Pubstage , "Final" ) ) And ( Limit-To ( Oa , "All" ) ) And ( Limit-To ( Pubyear , 2022 ) Or Limit-To ( Pubyear ,
2021 ) Or Limit-To ( Pubyear , 2020 ) Or Limit-To ( Pubyear ,
2019 ) Or Limit-To ( Pubyear , 2018 ) Or Limit-To ( Pubyear ,
2017 ) ) And ( Limit-To ( Doctype , "Ar" ) ) And ( Limit-To ( Subjarea , "Medi" ) ) And ( Limit-To ( Exactkeyword , "Informed Consent" ) ) And ( Limit-To ( Language , "English" ) ) And ( Limit-To ( Srctype , "J" ) )
|
Science Direct |
Title, abstract, keywords: comprehension
OR understanding OR knowledge AND adult AND patient AND surgical procedure
AND informed AND consent AND hospital |
Desired articles include published articles, English articles and full text articles. All articles are studies assessing patient understanding, factors influencing patient understanding and interventions to improve patient understanding of surgical informed consent. We excluded studies evaluating approval for clinical research, screening, diagnostics, and research into drug prescription; research on geriatric patients, unconscious patients and patients unable to give informed consent.�������������
Three reviewers evaluated the title and abstract of the article sought. After the search process, all selected articles will be collected and uploaded to the Mendeley Reference Manager and duplicates removed. Differences of opinion between reviewers regarding the data extraction process were resolved through discussion and the final decision was determined by agreement between the three. Data is presented in tabular form.�����
The search results and inclusion process of articles in this scoping review are described and presented according to the preferred reporting format for systematic review and meta-analysis extension for scoping review (PRISMA-ScR) flowchart.(Tricco et al., 2018) The prismatic study selection is shown in Figure 1.
Figure
1. PRISMA-ScR Research Diagram
Results
391 articles were identified in this
scoping review in the initial search. After removing duplication, language
barriers and full text only articles, a total of 127 articles were screened for
eligibility. After filtering the abstracts on the articles obtained, 102
articles were successfully excluded and 25 articles were found at the end of
the search. The articles obtained are summarized in table 3.
Articles included in this scoping review are composed of publications from 14 countries, starting from America,
Asia, Australia, Africa and Europe, but 57,1% of the articles (n=8) were from
the United States. Further investigation
into the types of studies revealed that most articles were observational
and experimental studies design, two studies were systematic review. The number
of participants in the articles obtained was quite varied, ranging from 10 to
420 patients. The characteristics of the articles in this scoping review are
presented in table 2.
Table 2.
Characteristics of The
Studies
Place |
|
America |
8 |
Australia |
2 |
Asia |
6 |
Europe |
5 |
Africa |
2 |
Methods |
|
Observational |
11 |
Experimental |
12 |
Systematic review |
2 |
Year of publications |
|
2017-2019 |
21 |
2020-2022 |
4 |
Participant |
|
<100 |
16 |
100-200 |
2 |
200-300 |
4 |
>300 |
3 |
��������������
11 articles determined the patient comprehension level of surgical informed consent and the influencing factors. Assessment of patient understanding was carried out after giving informed consent through a standard procedure, namely during the surgical informed consent process, the patient met and received important information about the surgical procedure from the surgeon verbally and then used that information to express consent and sign the operating agreement. After that, the patient was given a questionnaire to assess the patient understanding of the surgical consent. 4 articles were used structured questionnaire, 2 articles were used specific questionnaire, 2 articles were used open-ended questionnaire and 2 articles were used multiple choice questionnaire. Most of questionnaires were assessed by Likert scale. 6 articles evaluate patient understanding immediately, and 5 articles evaluate within days to months. 9 out of 11 articles (81.8%) stated that the patient comprehension of surgical informed consent was low. Factors affecting the level of understanding of those patients included education level, age and language limitations.
��� 14 articles compared the patient understanding of surgical informed consent with the standard informed consent process compared with the patient understanding using the addition of information media to the standard informed consent process. Additional information media used varied, ranging from written information media (pamphlets and leaflets), mobile applications, websites, and multimedia sources such as videos, cartoon illustrations or interactive presentations. Overall, 12 studies (85%) demonstrated that patient understanding had improved when using various additional information media in providing surgical consent.
Table 3
Selected Studies
Regarding Patient Comprehension Of Surgical Informed Consent.
No |
Title |
Country and year of
publication |
Study design |
Study aims |
Result |
|
1 |
Surgical
Decision-Making for Temporal Lobe Epilepsy: Patient Experiences of the
Informed Consent Process |
United
States, 2021 |
A
systematic analysis |
To
describe the informed process of anterior temporal lobectomy. |
Patients'
beliefs and expectations about treatment affect the consent process.
Decisions are often made before meeting with the surgeon |
|
2 |
Informed
consent for clinical treatment in low-income setting: evaluating the
relationship between satisfying consent and extent of recall of consent
information |
Nigeria,
2017 |
Cross
sectional |
To
evaluate a patient understanding of informed consent by assessing the extent
to which the patient can recall information believed to have been given |
78,3%
respondent could recall appropriately. A higher level of patient education
affects the comprehension level and ability to remember the information that
has been given. |
|
3 |
Effect
of Consent and Educational Adjuncts to Consent on Patient Perceptions About
Colonoscopy |
United
States, 2018 |
Cohort
|
To
assess intervention to improve the consent process. |
The
patient's ability to recall the information that has been given is increased
with the use of additional educational methods |
|
4 |
Development,
piloting, and evaluation of an evidence-based informed consent form for total
knee arthroplasty (evab-Pilot):
a protocol for a mixed methods study |
Germany,
2021 |
Mixed
methods |
To
evaluate the effects of evidence-based informed consent forms in comparison
with standard consent |
Evidence-based
and understandable presentation of risk in informed consent forms
strengthening the patients� competence |
|
5 |
Cultural
competency of a mobile, customized patient education tool for improving
potential kidney transplant recipients� knowledge and decision-making |
United
States, 2017 |
Multistage
analysis |
To
valuate the use of animated illustrations on mobile
applications to provide information about surgical procedures as a reference
for patients in making decisions |
The
use of animated images on mobile applications significantly provides a
reference for patients in making the right decisions |
|
6 |
Ethical
challenges of obtaining informed consent from surgical patients |
Iran,
2017 |
Descriptive
cross-sectional |
To
evaluate the effect of ethical challenges on the surgical informed consent
process |
Many
factors affect the low quality of informed consent, it is necessary to create
a framework to identify difficulties in the surgical informed consent process |
|
7 |
Visual
teaching aids improve patient understanding and reduce anxiety prior to a
colectomy |
United
States, 2020 |
A
randomized pilot study |
To
evaluate visual images as a medium of additional information on informed
consent for colectomy |
The
use of visual images during the consent process for a colectomy is useful for
helping patients recall surgical risk memories, reduce anxiety, and improve
understanding |
|
8 |
Patients
understanding of terminology commonly used during combined
orthodontic-orthognathic treatment |
United
Kingdom, 2019 |
Cross
sectional |
To
evaluate the effect of using medical terminology that is often used by
doctors on patient understanding |
The
use of medical terms commonly used by doctors cannot be understood, this
understanding can be influenced by differences in language and patient
education level |
|
9 |
Impact
of a multimedia support on the understanding of medical information by
hearing-impaired patients before cochlear implantation |
France,
2017 |
A
prospective single-centre single-blind study |
To
compare the use of verbal, written and digital forms of information
(�multimedia�) in surgical informed consent |
The
use of digital information media significantly improves patient
understanding, especially in patients with hearing loss |
|
10 |
Use
of Surgical Risk Preoperative Assessment System (SURPAS) and Patient
Satisfaction during Informed Consent for Surgery |
United
States, 2018 |
Cohort |
To
compare the effectiveness of using the Surgical Risk Preoperative Assessment
System (SURPAS) compared to the standard informed consent process |
The
SURPAS tool significantly improves patient understanding compared to the
standard consent process |
|
11 |
The
effectiveness of handout assisted versus verbal consent on post-operative
recall and understanding � A randomized control study |
Malaysia,
2017 |
Randomized
prospective study |
To
assess the effectiveness of using leaflet information media compared to verbal
information in improving patient understanding and the patient's ability to
recall information that has been given after surgery is performed |
Although
the use of leaflets can increase patient satisfaction, it does not
significantly increase patient understanding |
|
12 |
Informed
Consent for Patients Undergoing Transsphenoidal Excision of Pituitary
Adenoma: Development and Evaluation of a Procedure-Specific Online
Educational Resource |
United
Kingdom, 2018 |
Cohort
|
To
assess the effectiveness of using multimedia on the website as an interactive
medium in the approval process |
The
use of interactive multimedia information on multimedia websites can assist
in the informed consent process |
|
13 |
Improving
informed consent in cardiac surgery by enhancing preoperative education |
Australia,
2017 |
A
systematic review |
To
evaluate preoperative education in improving patient's memory and
understanding of surgical informed consent |
The
patient's ability to remember and understand information is significantly
improved with the use of various additional information media |
|
14 |
Assessing
adequacy of informed consent for elective surgery by student-administered
interview |
Singapore,
2018 |
A
prospective cross-sectional |
To
evaluate the quality of informed consent by assessing the patient
understanding and ability to recall the information provided |
Patient
understanding of surgical information and complications is generally low,
especially in geriatric patients |
|
15 |
Consent
for spine surgery: an observational study |
� |
A
prospective observational study |
To
analyze the effect of using videotape on the surgical informed consent
process |
The
patient is only able to understand part of the information provided by the surgeon |
|
16 |
Knowledge
and Perception of Ethiopian Surgical Patients to Informed Consent Practice
for Surgical Procedures |
Ethiopia,
2018 |
Cross-sectional
|
To
assess the patient's perception and understanding of surgical consent |
In
general, patients have limited knowledge, perception and understanding of
surgical informed consent |
|
17 |
Development
and pilot testing of an informed consent video for patients with limb trauma
prior to debridement surgery using a modified Delphi technique |
Taiwan,
2017 |
Pilot
study |
To
evaluate the use of educational videos on the knowledge, understanding and
satisfaction of patients with the informed consent process |
Educational
videos can be used to increase patient knowledge and satisfaction |
|
18 |
How
to effectively obtain informed consent in trauma patients: a systematic
review |
Taiwan,
2019 |
A
systematic review |
To
evaluate the use of various methods of providing information to improve
patients' understanding of surgical informed consent |
The
patient's memory and understanding of the risks of surgery both on the use of
written or video information media than if the information is given verbally |
|
19 |
Informed
consent and responses of surgical patients: A study in North India |
India,
2021 |
Cross-sectional
|
To
evaluate the patient's perspective and understanding of the contents of the
informed consent |
The
patient's level of awareness, perception and understanding of surgical
informed consent was found to be good |
|
20 |
Informed
consent through 3D virtual reality: a randomized clinical trial |
Italy,
2017 |
Prospective,
single-centre, randomized controlled clinical trial
|
To
evaluate the effect of using the new 3D anatomy visualizer in improving the
quality of communication between doctors and patients on the surgical
informed consent process |
3D
virtual reality significantly assists surgeons in establishing communication,
increasing patient understanding without increasing anxiety |
|
21 |
Lost
in translation: Informed consent in the medical mission setting |
United
States, 2017 |
Mixed-methods
study |
To
evaluate the information content provided by the surgeon, the patient's
understanding and the overall quality of informed consent |
The
patient has limited memory or understanding. Influenced by the surgeon's use
of language in providing information |
|
22 |
Patient
comprehension necessary for informed consent for vascular procedures is poor
and related to frailty |
United
States, 2018 |
Single
centre, prospective, cross- sectional study |
To
evaluate the patient's level of understanding of the basic information on
surgical informed consent |
The
patient understanding of the basic information provided by the surgeon is low |
|
23 |
Personalized
3D printed Model for Informed Consent for Stage I Lung Cancer: A Randomized
Pilot Trial |
Seoul,
2018 |
A
Randomized Pilot Trial |
To
evaluate the use of personalized 3D printed models to improve patient
understanding of informed consent for lung cancer resection surgery |
The
use of personalized 3D printing is technically feasible and has the potential
to improve patient understanding of informed consent |
|
24 |
Use
of an ipad application in preoperative counseling
for pelvic reconstructive surgery: a randomized trial |
United
States, 2018 |
A nonblinded, randomized, controlled trial |
To
evaluate the effect of using additional information media using the iPad
application on the informed consent process for pelvic organ prolapse surgery
(POP). |
Aplikasi ipad
berbasis POP tidak meningkatkan pemahaman pasien terhadap informed
consent bedah |
|
25 |
The
quality of informed consent obtained for orthopedic surgeries� elective
versus trauma: A prospective interview-based study |
Israel,
2018 |
A
prospective interview-based cohort study |
To
evaluate the quality of the informed consent process in emergency surgery in
trauma patients versus elective orthopedic surgery |
Patient
understanding of emergency surgery in trauma patients is lower than in
elective orthopedic surgery |
Discussion
Patient comprehension on surgical informed consent
The
purpose of this scoping review is to evaluate the patient understanding of
surgical informed consent. The reviewers identified twenty five studies
for analysis and found that most of the articles mentioned that patients had
poor comprehension of surgical informed consent that has been given. The
patient's low understanding of surgical informed consent is not only obtain in
elective surgery patients but also in trauma patients undergoing emergency
surgery, these results are listed in a study conducted by Shemesh.(Shemesh et al., 2019)
The lack of patient understanding is a
worrying phenomenon. Only two articles in this scoping review mentioned a good
patient comprehension level in the studies they had done. Nnabugwu
et al, in their research found that 78.3% of respondents have the ability to
remember information. The ability to recall information explained by doctors
during the surgical informed consent process illustrates the level of patient
understanding of a good.(Nnabugwu, Ugwumba, Udeh, Anyimba, & Ozoemena,
2017) Singh et al also
explained the same thing, in their study it was stated that the patient's level
of understanding of surgical informed consent was good.(Singh, Rochwani, & Oberoi, 2021).
Factors affecting patient comprehension on surgical
informed consent
There were many obstacles in providing
optimal informed consent in medical practice.(Overcarsh, Arvizo,
& Harvey, 2019) This
caused the patient's understanding of surgical informed consent was still low.
It is not easy for a surgeon to provide information to the patient and make the
patient understand what is being explained, but this must be done. It is
important to describe the causes of the patient's low level of understanding so
that we can arrange strategies to deal with each obstacle.
In order to understand the reasons for
the patient's poor understanding of surgical informed consent, we need to
review essential aspects of the surgical informed consent process, which the
patient understanding of important information in surgical informed consent, including
the risks, benefits of surgery, and alternative surgical procedures; patient
competence; and patient consent given without coercion.(Lin et al., 2019; Overcarsh et al., 2019) A good informed consent process is when the
patient is able to receive, understand and recall important information
provided by the doctor and use that information as a reference in giving
voluntary consent. The process is formed from adequate communication between
the patient and the surgeon and it is very important to be done before the
operation.
An
important things that must be conveyed in the communication and discussion between doctors and
patient are expected course of illness; proposed
surgical intervention; morbidity and mortality associated with proposed
surgical intervention; common complications of surgery; risk and benefits of proposed
operation; perioperative expectations; alternative treatment options;
disclosure of surgical team members and role in surgical care.(Overcarsh et al., 2019) All those information must be conveyed by the surgeon to the patient to
form a good understanding, but some doctors fail to convey information properly
and many patients are unable to remember the diagnosis, surgical procedure,
risks and benefits of surgery and complications.(Lin et al., 2019)
Ruske et al suggested
that one possible explanation for the patient's low understanding is the
inadequate information and education provided by the surgeon. They have found that patients were often not fully informed about their procedure even if patient are capable to understand.(Ruske et al., 2021) Surgeon were frequently unaware that in the informed consent process, a
relationship of disclosure and trust is built with the patient and promote
mutual understanding to reduce unnecessary litigation in unexpected event.(Lemmu, Megersa, Abebe, & Abebe, 2020; Ruiz L�pez,
2013) Besides, not all
surgeons are aware of the importance of providing medical information to
patients in their clinical work. Some surgeons believe that because all the
information provided is included in the surgical consent, there is no need to
explain to the patient verbally the details of the procedure. But
unfortunately, as mentioned in the study conducted by Lemmu
et al, many patients did not read the informed consent letter before signing
it.(Lemmu et al., 2020; Lin et al., 2019; Ruiz L�pez,
2013)
Education level and age plays a major
role in the ability of patients to understand the information presented to
them.(Garc�a-Garc�a, S�nchez-Sabater, Medrano-Morte, &
Luna-Maldonado, 2019; Ruske et al., 2021) The low level of patient education as
the cause of the patient's low understanding of surgical informed consent was
found in the study conducted by Sceats et al,
therefore it is important to convey information in an easy-to-understand way.(Sceats et al., 2019) Shemesh
et al also showed a correlation between the comprehension of informed consent
and the level of education, with a significantly better comprehension in
patients with post-high-school education.(Shemesh et al., 2019) Nnabugwu et al revealed that patient might not be
unconnected when surgeon provide the information.(Nnabugwu et al., 2017) In the study of Moeini
et al, it was found that the higher the education level of the patients, the
more they are willing to receive information. Information to patients should be
based on their level of education, and presenting the same information to all
patients is inappropriate.(Moeini et al., 2020)
Chia et al stated that the surgical
informed consent process in geriatric patients is more complicated because
geriatric patients have many comorbidities, weakness and even cognitive
impairment. A different approach to providing informed consent by surgeons is
needed, more time and repeated reinforcement is needed.(Chia, Chan, Ng, Rao, & Singaporewalla, 2019) Zimmermann et al found a strong correlation
between patient understanding and age, in that study it was stated that young
people (aged up to 39 years) were significantly more aware of the surgical
informed consent given.(Zimmermann et al., 2021) A different result in this review is the study by Nnabugwu et al. which stated that there was no difference
in the level of understanding among young people compared to older people. This
is due to the limited access of our younger patients to other appropriate
sources of medical information including the internet which makes younger and
older patients share a common knowledge base.(Nnabugwu
et al., 2017)
A study by Garc�a-Garc�a et al revealed
that education level was significantly related to gender and age. They found
that the percentage of women who were uneducated was higher than men, and the
percentage of women studying at university was twice than men.(Garc�a-Garc�a et al., 2019) This gender difference is related to
the opportunity for men to pursue higher education.
Various studies mentioned that
limitation in language and surgeon communication skill were the factors that
affects the patient's level of understanding.(Sceats et al., 2019) Differences in language and the complexity of medical terms used by
surgeons are obstacles in the communication process. Research by Benning et al.
evaluated that the specific terms used to describe medical concepts, diagnoses
and most of the terms commonly used by doctors were poorly understood by
patients, not easy for patients to understand medical terminology.(Benning, Madadian,
Seehra, & Fan, 2021) In addition, Shemesh et al stated that the complete
informed consent process was time consuming. very likely the patient loses
attention when the doctor gives information and ignores important things.
Patients are generally only able to understand "basic" elements
rather than more complex information.(Shemesh et al., 2019) Li Ching Ng et al in their research said that the
surgeon's inconsistency in explaining the information needed to obtain informed
consent was the cause of the low level patient comprehension of surgical
informed consent.(Li Ching Ng et al., 2021)
Therefore,
improvement of communication quality is highly important
for all surgeons. Surgeons need to enhance
their communication skills, use structured conversation, choose words more
carefully and maintain consistency in providing information.(Agozzino et al., 2019) The use of terms that are easily understood by patients
as well as effective use of time will increase the validity of informed
consent.(Benning et al., 2021;
Shemesh et al., 2019) To make all that easier, can be considered other means of
communication in giving informed consent.
Improving
patient comprehension on surgical
informed consent
The information submitted by the
doctor verbally in the standard informed consent process fails to provide a
good understanding to the patient. Several strategies are used to promote
better patient understanding, including use of additional information media
after verbal information. It is stated that the patient's ability to remember
and understand the information given before surgery is significantly improved
with the use of various educational tool. The use of additional information
media allows patients to better understand medical information and becomes the
basis of reference in the decision-making process.(Paterick, Paterick, & Paterick, 2020)
Most studies use written information
as addition to the discussion. Written information sheets are very useful
because they can be given before the day of surgery, enabling patients and
families to review the information they need repeatedly. There are many studies
that aim to find out whether the use of written information can improve patient
comprehension. One of them is the study of Zimmermann et al which stated that
there are significant results on patient understanding using written informed
consent.(Zimmermann et al., 2021) However
the article found in this scoping review contradicts some of the studies
above. Kong et al found
that the use of handouts as written information was not significant help the
patient to remember.(Koong, Rajandram, Sidambram, & Narayanan, 2021) This statement is supported by
research by Agozzino et al which stated that the
written informed consent forms are not sufficient to provide information and
make the patients understand of choices they made for their health.(Agozzino et al., 2019)
Three reasons underlie those findings:
first, many patients showed infrequent interest in informed consent documents.
Patients with a high level of education and less than 60 years are more likely
to read consent forms.(Agozzino et al., 2019) Wong et al in their research in
Canada confirmed that written information media that was read and understood by
patients could improve patient understanding. Patient motivation and the
ability to understand this information are important factors that play a role
in it.(Wong et al., 2016) Second, the surgeon may show a lack
of interest in the document, seeking only the signature without providing
adequate motivation to the patient for the importance of reading the document.
Third, leaflets were given on the day of consent or even just a few hours
before the surgical procedure, thereby not allowing enough time for the patient
to fully appreciate the information.(Agozzino et al., 2019)
It can be concluded from
studies of Kong et al that written
informed consent alone is not sufficient. With advances in technology, there
are many information media that can be used as additional tools in the informed
consent process. Three articles found in the scoping review demonstrated the
use of visual aids such as cartoon illustrations or other 3D images that could
potentially improve patients' understanding of surgical informed consent. The
use of colour images attracts patients to read and
makes it easier for patients to understand surgical procedures that are
difficult for ordinary people to understand.(Perin et al., 2021; Stewart et al., 2021; Yoon et
al., 2019)
Current technological advances offer
many multimedia applications that can be used to simplify the health care
process, including the process of giving informed consent. There were 6 articles in this scoping review use audio-visual
presentations, mobile applications and other interactive educational websites
as tools to provide information about surgical procedures and get significant
results in improving the level of patient comprehension.(Axelrod et al., 2017;
Fudman et al., 2019; Lin et al., 2017; Marcus, Jain, Grieve, & Dorward,
2018; Rouf et al., 2017; Weise et al., 2021; Wiesen et al., 2020) The use of interesting audio visuals provides
answers for patients with physical limitations. Rouf
et al stated that digital support improves the understanding of patients with
hearing loss in cochlear implantation surgery (Rouf et al., 2017).
Interesting and interactive information media can be used in the
informed consent process, but good verbal communication is an important part
and should not be replaced by other forms of communication (Kinman et al., 2018).
Conclusion ���
Numerous articles review the patient's understanding of
surgical informed consent. From the results obtained, the patient's level of
understanding is still low. It is not easy to create a well informed consent
process in a surgical setting. Many factors influence patient understanding
including the surgeon's level of education and communication skills. Providing
adequate, structured and comprehensive education in the preoperative
communication process to patients is very important. Several informational
methods can be used to improve the patients
comprehension of surgical informed consent.
BIBLIOGRAPHY
Agozzino,
Erminia, Borrelli, Sharon, Cancellieri, Mariagrazia, Carfora, Fabiola Michela,
Di Lorenzo, Teresa, & Attena, Francesco. (2019). Does written informed
consent adequately inform surgical patients? A cross sectional study. BMC
Medical Ethics, 20(1), 4�11.
https://doi.org/10.1186/s12910-018-0340-z.
Alvionita, Vinny, Harahap, Juliandi, & Aini, Nur. (2021).
Analysis of the Implementation of Informed Consent. Journal of Healthcare
Technology and Medicine, 7.
Axelrod, David A., Kynard-Amerson, Crystal S., Wojciechowski,
David, Jacobs, Marie, Lentine, Krista L., Schnitzler, Mark, Peipert, John D.,
& Waterman, Amy D. (2017). Cultural competency of a mobile, customized
patient education tool for improving potential kidney transplant recipients�
knowledge and decision-making. Clinical Transplantation, 31(5),
1�9. https://doi.org/10.1111/ctr.12944.
Bazzano, Lydia A., Durant, Jaquail, & Brantley, Paula
Rhode. (2021). A modern history of informed consent and the role of key
information. Ochsner Journal, 21(1), 81�85.
https://doi.org/10.31486/toj.19.0105.
Benning, Amanveer, Madadian, Matin Ali, Seehra, Jadbinder,
& Fan, Kathleen. (2021). Patients understanding of terminology commonly
used during combined orthodontic-orthognathic treatment. Surgeon, 19(5),
e193�e198. https://doi.org/10.1016/j.surge.2020.09.012.
Chia, Clement L. K., Chan, Kai Siang, Ng, Marcus J. M., Rao,
Anil D., & Singaporewalla, Reyaz. (2019). Assessing adequacy of informed
consent for elective surgery by student-administered interview. ANZ Journal
of Surgery, 89(6), 677�682. https://doi.org/10.1111/ans.15214.
Cocanour, Christine S. (2017). Informed consent�It�s more
than a signature on a piece of paper. American Journal of Surgery, 214(6),
993�997. https://doi.org/10.1016/j.amjsurg.2017.09.015
Dewar, Sandra R., Pieters, Huibrie C., & Fried, Itzhak.
(2021). Surgical Decision-Making for Temporal Lobe Epilepsy: Patient
Experiences of the Informed Consent Process. Frontiers in Neurology, 12(December),
1�12. https://doi.org/10.3389/fneur.2021.780306.
Fink, Aaron S., Prochazka, Allan V., Henderson, William G.,
Bartenfeld, Debra, Nyirenda, Carsie, Webb, Alexandra, Berger, David H., Itani,
Kamal, Whitehill, Thomas, Edwards, James, Wilson, Mark, Karsonovich, Cynthia,
& Parmelee, Patricia. (2010). Predictors of Comprehension during Surgical
Informed Consent. Journal of the American College of Surgeons, 210(6),
919�926. https://doi.org/10.1016/j.jamcollsurg.2010.02.049.
Fudman, David I., Papamichael, Konstantinos, Roemi, Lilach,
Rao, Varun, Falchuk, Kenneth R., Leffler, Daniel A., & Feuerstein, Joseph
D. (2019). Effect of Consent and Educational Adjuncts to Consent on Patient
Perceptions about Colonoscopy. Journal of Clinical Gastroenterology, 53(8),
e316�e321. https://doi.org/10.1097/MCG.0000000000001093.
Garc�a-Garc�a, E. M., S�nchez-Sabater, B., Medrano-Morte, I.,
& Luna-Maldonado, A. (2019). Sociodemographic factors affecting the
comprehension of clinical information by inpatients undergoing trauma surgery. Revista
Espa�ola de Cirug�a Ortop�dica y Traumatolog�a (English Edition), 63(5),
355�360. https://doi.org/10.1016/j.recote.2019.04.005.
Hall, Daniel E., Morrison, Penelope, Nikolajski, Cara, Fine,
Michael, Arnold, Robert, & Zickmund, Susan L. (2012). Informed consent for
inguinal herniorrhaphy and cholecystectomy: Describing how patients make
decisions to have surgery. American Journal of Surgery, 204(5),
619�625. https://doi.org/10.1016/j.amjsurg.2012.07.020.
Herfiyanti, Leni. (2015). Kelengkapan Informed Consent
Tindakan Bedah Menunjang Akreditasi Jci Standar Hpk 6 Pasien Orthopedi. Jurnal
Manajemen Informasi Kesehatan Indonesia, 3(2), 81�88.
https://doi.org/10.33560/.v3i2.89.
Khan, Robyna Irshad. (2012). Patient Safety and the
Importance of Informed Consent. In Urolithiasis: Basic Science and Clinical
Practice (pp. 543�546). https://doi.org/10.1007/978-1-4471-4387-1_68.
Khoshrang, Hossein, Taramsari, Morteza Rahbar, Alavi, Cyrus
Emir, Soleimani, Robabeh, Rimaz, Siamak, Sedighinejad, Abbas, Habibi, Mohammad
Reza, & Bayat, Zoleikha. (2021). Research Paper: The Quality of Informed
Consent Obtaining Before Electroconvulsive Therapy: A Report From a Referral,
Academic Hospital. Caspian Journal of Neurological Sciences, 7(1),
37�41. https://doi.org/10.32598/CJNS.7.24.6.
Kinman, Casey L., Meriwether, Kate V., Powell, Cayse M.,
Hobson, Deslyn T. G., Gaskins, Jeremy T., & Francis, Sean L. (2018). Use of
an iPadTM application in preoperative counseling for pelvic
reconstructive surgery: a randomized trial. International Urogynecology
Journal, 29(9), 1289�1295. https://doi.org/10.1007/s00192-017-3513-2.
Koong, Jun Kit, Rajandram, Retnagowri, Sidambram, Naveendran,
& Narayanan, Vairavan. (2021). The effectiveness of handout assisted versus
verbal consent on post-operative recall and understanding - A randomized
control study. Surgeon, (xxxx), 1�8.
https://doi.org/10.1016/j.surge.2021.04.002.
Lemmu, Befekadu, Megersa, Abebe, Abebe, Engida, & Abebe,
Kirubel. (2020). Knowledge and Perception of Ethiopian Surgical Patients to
Informed Consent Practice for Surgical Procedures. Open Access Surgery, 13,
65�70. https://doi.org/http://dx.doi.org/10.2147/OAS.S268009.
Li Ching Ng, Angela, McRobb, Lucinda S., White, Sarah J.,
Cartmill, John A., Cyna, Allan M., & Seex, Kevin. (2021). Consent for spine
surgery: an observational study. ANZ Journal of Surgery, Vol. 91, pp.
1220�1225. https://doi.org/10.1111/ans.16348.
Lin, Yen Ko, Chen, Chao Wen, Lee, Wei Che, Lin, Tsung Ying,
Kuo, Liang Chi, Lin, Chia Ju, Shi, Leiyu, Tien, Yin Chun, & Cheng, Yuan
Chia. (2017). Development and pilot testing of an informed consent video for
patients with limb trauma prior to debridement surgery using a modified Delphi
technique. BMC Medical Ethics, 18(1), 1�12.
https://doi.org/10.1186/s12910-017-0228-3.
Lin, Yen Ko, Liu, Kuan Ting, Chen, Chao Wen, Lee, Wei Che,
Lin, Chia Ju, Shi, Leiyu, & Tien, Yin Chun. (2019). How to effectively
obtain informed consent in trauma patients: A systematic review. BMC Medical
Ethics, 20(1). https://doi.org/10.1186/s12910-019-0347-0.
Marcus, Hani J., Jain, Abhiney, Grieve, Joan, & Dorward,
Neil L. (2018). Informed Consent for Patients Undergoing Transsphenoidal
Excision of Pituitary Adenoma: Development and Evaluation of a
Procedure-Specific Online Educational Resource. World Neurosurgery, 118,
e933�e937. https://doi.org/10.1016/j.wneu.2018.07.102.
Moeini, Sanaz, Shahriari, Mohsen, & Shamali, Mahdi.
(2020). Ethical challenges of obtaining informed consent from surgical
patients. Nursing Ethics, 27(2), 527�536. https://doi.org/10.1177/0969733019857781.
Mudiyanselage, Udagedara, Eshana, Jayami, Mathangasinghe,
Yasith, Sarath, Anura, & Banagala, Kumara. (2019). Are predominantly
western standards and expectations of informed consent in surgery applicable to
all ? A qualitative study in a tertiary care hospital in Sri Lanka.
https://doi.org/10.1136/bmjopen-2018-025299.
Nnabugwu, Ikenna I., Ugwumba, Fredrick O., Udeh, Emeka I.,
Anyimba, Solomon K., & Ozoemena, Oyiogu F. (2017). Informed consent for
clinical treatment in low-income setting: Evaluating the relationship between
satisfying consent and extent of recall of consent information. BMC Medical
Ethics, 18(1), 1�8. https://doi.org/10.1186/s12910-017-0227-4.
Nuraeni, RIna. (2016). Relationship of surgical procedure
knowledge with level anxiety of adult surgical patients in the surgical room of
cideres hospital period of May-June 2015. Journal of Nursing and Health,
II.
Overcarsh, Patricia, Arvizo, Cynthia, & Harvey, Lara.
(2019). Informed consent in gynecologi surgery. Current Opinion in
Obstetrics and Gynecology, 31(4), 240�244.
https://doi.org/10.1097/GCO.0000000000000550.
Pallett, Alicia C., Nguyen, Bao T., Klein, Natalie M.,
Phippen, Neil, Miller, Caela R., & Barnett, Jason C. (2018). A randomized
controlled trial to determine whether a video presentation improves informed
consent for hysterectomy. American Journal of Obstetrics and Gynecology,
219(3), 277.e1-277.e7. https://doi.org/10.1016/j.ajog.2018.06.016.
Paterick, Zachary R., Paterick, Timothy Edward, &
Paterick, Barb Block. (2020). Medical informed choice: Understanding the
element of time to meet the standard of care for valid informed consent. Postgraduate
Medical Journal, 96(1141), 708�710.
https://doi.org/10.1136/postgradmedj-2019-137278.
Perin, Alessandro, Galbiati, Tommaso Francesco, Ayadi,
Roberta, Gambatesa, Enrico, Orena, Eleonora Francesca, Riker, Nicole Irene,
Silberberg, Hagit, Sgubin, Donatella, Meling, Torstein Ragnar, & DiMeco,
Francesco. (2021). Informed consent through 3D virtual reality: a randomized
clinical trial. Acta Neurochirurgica, 163(2), 301�308.
https://doi.org/10.1007/s00701-020-04303-y.
Pucher, Philip H., Johnston, Maximilian J., Archer,
Stephanie, Whitby, Jack, Dawson, Peter M., Arora, Sonal, & Darzi, Ara.
(2019). Informing the Consent Process for Surgeons: A Survey Study of Patient
Preferences, Perceptions, and Risk Tolerance. Journal of Surgical Research,
235(0), 298�302. https://doi.org/10.1016/j.jss.2018.09.046.
Richardson, Victoria. (2013). Patient comprehension of
informed consent. Journal of Perioperative Practice, 23(1�2),
26�30. https://doi.org/10.1177/1750458913023001-204.
Rouf, C. E., Lescanne, E., Villeneuve, A., Reffet, K., Kim,
S., & Bakhos, D. (2017). Impact of a multimedia support on the understanding
of medical information by hearing-impaired patients before cochlear
implantation. European Annals of Otorhinolaryngology, Head and Neck Diseases,
134(6), 387�392. https://doi.org/10.1016/j.anorl.2017.05.001.
Ruiz L�pez, Pedro. (2013). Informed Consent in Surgery.
Distance Between Theory and Practice. Cirug�a Espa�ola (English Edition),
91(9), 551�553. https://doi.org/10.1016/j.cireng.2013.12.015.
Ruske, Jack, Sharma, Gaurav, Makie, Kevin, He, Katherine,
Ozaki, C. Keith, Menard, Matthew T., Belkin, Michael, & Shah, Samir K.
(2021). Patient comprehension necessary for informed consent for vascular
procedures is poor and related to frailty. Journal of Vascular Surgery, 73(4),
1422�1428. https://doi.org/10.1016/j.jvs.2020.06.131.
Sceats, Lindsay A., Morris, Arden M., Narayan, Raja R.,
Mezynski, Ana, Woo, Russell K., & Yang, George P. (2019). Lost in
translation: Informed consent in the medical mission setting. Surgery
(United States), 165(2), 438�443.
https://doi.org/10.1016/j.surg.2018.06.010.
Shemesh, S., Sidon, E., Heller, S., Cohen N, Kosashvili, Y.,
Dovrat, R., Velkes, S., & Burg, A. (2019).
The_quality_of_informed_consent_obtained_for_ortho.pdf. Journal
OfOrthopaedic Surgery 27(2), 27(2), 1�8.
Singh, Aman, Rochwani, Ritu, & Oberoi, Simmi. (2021). Informed
consent and responses of surgical patients: A study in North India. National
Journal of Physiology, Pharmacy and Pharmacology, 11(8), 1.
https://doi.org/10.5455/njppp.2021.11.06217202109072021.
Stewart, James A., Wood, Lauren, Wiener, Jameson, Kennedy,
Gregory D., Chu, Daniel I., Lancaster, Jeffrey R., & Morris, Melanie S.
(2021). Visual teaching aids improve patient understanding and reduce anxiety
prior to a colectomy. American Journal of Surgery, 222(4),
780�785. https://doi.org/10.1016/j.amjsurg.2021.01.029.
Susanto, Dewangga Primananda, Pratama, Bramantya Surya, &
Hariyanto, Tuti. (2017). Analysis of Factors Affecting Patient Understanding of
Informed Consent in Hospitals. Indonesian Journal of Health Management, 5(2),
73�81. https://doi.org/10.14710/jmki.5.2.2017.73-81.
Tricco, Andrea C., Lillie, Erin, Zarin, Wasifa, O�Brien,
Kelly K., Colquhoun, Heather, Levac, Danielle, Moher, David, Peters, Micah D.
J., Horsley, Tanya, Weeks, Laura, Hempel, Susanne, Akl, Elie A., Chang,
Christine, McGowan, Jessie, Stewart, Lesley, Hartling, Lisa, Aldcroft, Adrian,
Wilson, Michael G., Garritty, Chantelle, Lewin, Simon, Godfrey, Christina M.,
MacDonald, Marilyn T., Langlois, Etienne V., Soares-Weiser, Karla, Moriarty,
Jo, Clifford, Tammy, Tun�alp, �zge, & Straus, Sharon E. (2018). PRISMA
extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals
of Internal Medicine, 169(7), 467�473.
https://doi.org/10.7326/M18-0850.
Ulfa, Henny Maria. (2018). Analisa Kelengkapan Informed
Consent Tindakan Operasi di Rumah Sakit Sansani Pekanbaru. Jurnal INOHIM,
6(1), 21�26.
Villanueva, Claudia, Talwar, Arpit, & Doyle, Mathew.
(2018). Improving informed consent in cardiac surgery by enhancing preoperative
education. Patient Education and Counseling, 101(12), 2047�2053.
https://doi.org/10.1016/j.pec.2018.06.008.
Weise, Alina, L�hnen, Julia, B�hn, Stefanie, Steffen,
Felicia, Zacher, Sandro, Lauberger, Julia, Ates, Deha Murat, B�hmer, Andreas,
Rosenau, Henning, Steckelberg, Anke, & Mathes, Tim. (2021). Development,
piloting, and evaluation of an evidence-based informed consent form for total
knee arthroplasty (EvAb-Pilot): a protocol for a mixed methods study. Pilot
and Feasibility Studies, 7(1), 1�11. https://doi.org/10.1186/s40814-021-00843-x
Wiesen, Brett M., Bronsert, Michael R., Aasen, Davis M.,
Singh, Abhinav B., Lambert-Kerzner, Anne, Henderson, William G., Hammermeister,
Karl E., & Meguid, Robert A. (2020). Use of Surgical Risk Preoperative
Assessment System (SURPAS) and Patient Satisfaction During Informed Consent for
Surgery. Journal of the American College of Surgeons, 230(6),
1025-1033.e1. https://doi.org/10.1016/j.jamcollsurg.2020.02.049.
Wong, Alison L., Martin, Janet, Tang, David, Leblanc, Martin,
Morris, Steven F., Paletz, Justin, Stein, John, Wong, Michael J., &
Bezuhly, Michael. (2016). The Effect of Written Information on Recall of
Surgical Risks of Carpal Tunnel Release Surgery: A Randomized Controlled Study.
Plastic and Reconstructive Surgery, 138(6), 1011e-1018e.
https://doi.org/10.1097/PRS.0000000000002771.
Yoon, Soon Ho, Park, Samina, Kang, Chang Hyun, Park, In Kyu,
Goo, Jin Mo, & Kim, Young Tae. (2019). Personalized 3D-Printed Model for
Informed Consent for Stage I Lung Cancer: A Randomized Pilot Trial. Seminars
in Thoracic and Cardiovascular Surgery, 31(2), 316�318.
https://doi.org/10.1053/j.semtcvs.2018.10.017.
Zimmermann, Agnieszka, Pilarska, Anna,
Gaworska-Krzemińska, Aleksandra, Jankau, Jerzy, & Cohen, Marsha N.
(2021). Written informed consent�translating into plain language. A pilot
study. Healthcare (Switzerland), 9(2).
https://doi.org/10.3390/healthcare9020232.
������
Copyright holder: Rieke Cahya, Diah Melly
Maryana, Adik Wibowo (2022) |
First publication right: Syntax Literate: Jurnal Ilmiah Indonesia |
This article is licensed under: |